 | View Full Version : robot.txt Crazy BigGaz 03-08-2002, 09:57 AM Hi Does anyone know where the robot.txt file is loctated on RaQ 3 and RaQ 4 servers ? as I can't find it and if it doesn't have one can I make one if so how ? Thanks Garry jjma 03-08-2002, 10:21 AM I don't think there is one? If this is for just stopping search engines you could ssh into your machine, go to the directory that needs the robots.txt and type: touch robots.txt Jjma blacknight 03-09-2002, 08:08 AM Quite a few of the search engines look for robots.txt, so if you don't add one to a domain you'll get a lot of 403s in the logs. Crazy BigGaz 03-09-2002, 10:53 AM Hi So do I just put in a robot.txt file with nothing in it ? Thanks Garry blacknight 03-09-2002, 11:00 AM You could put in an empty file, but it wouldn't be that useful. Robots FAQ (http://www.robotstxt.org/wc/faq.html) gives a helpful introduction to what can be done. If the search engine robot is NOT a rogue, you can tell it which areas of your site to visit and which to keep out of... There are a few free/shareware utilities to help with the syntax if you are having problems. |