Results 1 to 2 of 2
  1. #1
    Join Date
    Feb 2004
    Phoenix, AZ

    Do searchbots respect your fast network connection?

    Some searchbots can get pretty aggressive, have you had to restrict them sometimes on a fast network connection?

  2. #2
    Join Date
    Dec 2006
    You can add the following to your robots.txt file to set the delay between successive hits to at least N seconds for Bing and Yahoo. Others may also take notice of it.

    User-Agent: *
    Crawl-delay: N

    For Google you can sign up at and manage various aspects of how your sites are indexed including the crawling rate.

Similar Threads

  1. Replies: 0
    Last Post: 10-23-2008, 04:40 AM
  2. Replies: 0
    Last Post: 04-30-2008, 05:42 PM
  3. Wireless Network Connection; Connection Failed!
    By Wojtek in forum Web Hosting Lounge
    Replies: 10
    Last Post: 03-27-2006, 11:57 AM
  4. Unmetered Ethernet Connection and Fast Ethernet Connection
    By inteltechs in forum Colocation and Data Centers
    Replies: 2
    Last Post: 12-29-2004, 06:10 PM
  5. complex templates and searchbots
    By ciderbob in forum Web Design and Content
    Replies: 0
    Last Post: 08-03-2004, 01:29 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts