Results 1 to 11 of 11
  1. #1

    Robot.txt file is cached by Google or not?

    I just want to no whether Google indexed Robot.txt file or not
      0 Not allowed!

  2. #2
    no not at all google doesn't crawl robot.txt pages
      0 Not allowed!

  3. #3
    Join Date
    Nov 2013
    Google doesn't crawl the robot.txt pages where the status is nofollow. If u want to hide the pages from Google crawler just put the status to "nofollow" to that page.
      0 Not allowed!

  4. #4
    Quote Originally Posted by aycomponent View Post
    I just want to no whether Google indexed Robot.txt file or not
    Everyone below is say'n google does not crawl robot.txt but the answer is that google crawler does look for robot.txt at first place and look for instructions in it to crawl the rest of the webpages. Robot.txt is not indexed.
      0 Not allowed!

  5. #5
    robots.txt are not cached but if something is disallowed in robots.txt and then removed it can take Google 48hrs to reflect this change and index the page in question.
      0 Not allowed!

  6. #6

    I don't think that Google is crawling robots.txt file or url. Because Robots.txt is a text file which is used to tell the search engine to don't crawl those pages or part of site which is mentioned on that list. Some time it happens that page is already cached or indexed by search engine before inserting in robots.txt and we think robots is not working so its a rumor and nothing
      0 Not allowed!

  7. #7
    Google does not crawl a Robots.txt file because in that sections using a format disallow.. so Google Crawler is not fatch that code..
      0 Not allowed!

  8. #8
    No Mate Google doesn't crawl Robot.txt file.
      0 Not allowed!

  9. #9
    Sorry to bother, but regarding the robots.txt, whats the command to prevent only google bots from crawling all pages on the server?
      0 Not allowed!

  10. #10
    Google or search engine spider are read robot.txt file and their instructions then allow and disallows those instruction. This is only generate for search engine.
      0 Not allowed!

  11. #11

    robots.txt is crawled by google

    robots.txt is crawled by Google. With the help of this file only, crawler knows which pages are not allowed to crawl.
    If you want only Google bot not to crawl your all pages but all other search engines should crawl, this is the code you have to write in robots.txt file:

    user-agent: Google

    By writing this,only Google will not crawl your website but other search engines will do.
      0 Not allowed!

Similar Threads

  1. How to stop a file from being read in robot.txt
    By bambinou in forum Programming Discussion
    Replies: 4
    Last Post: 08-13-2010, 07:44 AM
  2. advice for robot.txt file...
    By indiaberry in forum SEO / SEM Discussions
    Replies: 4
    Last Post: 06-27-2007, 10:05 AM
  3. robot.txt
    By GazCBG in forum Dedicated Server
    Replies: 4
    Last Post: 03-09-2002, 11:00 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts