Results 1 to 6 of 6
  1. #1
    Join Date
    Oct 2013
    Posts
    53

    Question Robots.txt question

    I see in my robots txt with these texts.
    Could you explain whats the meaning of this code?

    Code:
    User-agent: Fasterfox
    Disallow: /
    Crawl-delay: 10
    Is there anything blocks or its looks like fine?

  2. #2
    Join Date
    May 2012
    Location
    India
    Posts
    1,026

  3. #3
    Join Date
    May 2013
    Location
    India
    Posts
    748
    Yes, it will tell fasterbox user agent not to crawl any pages of that website. I don't what is fasterbox is; I haven't seen it yet. But if it is a malicious bot, there is chances that it do not respect the robots.txt file and can ignore the rules. Also I have seen somewhere that Crawl-delay is not an SEO friendly command, and do not recognize by some well known crawlers ( like google ). So avoid using it for general crawl rules.

  4. #4
    Join Date
    May 2009
    Location
    India
    Posts
    150

    Robots.txt

    Quote Originally Posted by neoearnmoney View Post
    I see in my robots txt with these texts.
    Could you explain whats the meaning of this code?

    Code:
    User-agent: Fasterfox
    Disallow: /
    Crawl-delay: 10
    Is there anything blocks or its looks like fine?

    As other members have said, it Disallow Fasterfox bot to stop crawling your website. Like this you can give instructions to bots you don't want to crwal your website. Also, you can set instruction to crwal your website with
    User-agent: *
    Disallow:
    Mobile App Development Company
    Mobile Programming for iOs, Android, Windows Phone, Blackberry

  5. #5
    Join Date
    Mar 2013
    Posts
    1,294
    Mind you, bot might have been designed in such a way that it doesn't give a damn about what's in the robots file.

  6. #6
    Firefox users use an extension called “Fasterfox”. This extension “pre-fetches” links on a page so that if the user were to click on a link it would load much faster because its already been downloaded. This may be more convenient for viewer, but is a major problem for many webmasters who are low on bandwidth.

    Source: google search

Similar Threads

  1. Which is better Robots.txt or Meta Robots Tag?
    By nitsingh in forum SEO / SEM Discussions
    Replies: 10
    Last Post: 03-01-2013, 04:08 PM
  2. Robots.txt question.
    By Vcandle in forum Web Design and Content
    Replies: 7
    Last Post: 11-07-2007, 10:37 AM
  3. robots.txt help
    By grabmail in forum Programming Discussion
    Replies: 4
    Last Post: 08-04-2006, 05:22 PM
  4. what is robots.txt
    By aamir-designer in forum Web Design and Content
    Replies: 2
    Last Post: 05-23-2006, 06:47 PM
  5. Directing Robots With Robots.txt
    By Nullified in forum Web Design and Content
    Replies: 0
    Last Post: 02-27-2005, 12:12 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •