How do I do it? I know I can make a robots.txt file but have heard that often robots don't bother listening to it. Is there a way to do this in Apache?

Originally posted by daveman
How do I do it? I know I can make a robots.txt file but have heard that often robots don't bother listening to it. Is there a way to do this in Apache?

Many different robots need different commands to specifically exclude them here is a list of commonly known robots http://www.robotstxt.org if they dont look for robots.txt consider it rogue and block the ip