Results 1 to 12 of 12
  1. #1

    Google stop crawl my site

    My URL is: <<removed>>

    Google stop crawl my site, it said robots.txt unreachable, but i open it on my browser, it's ok,and then i move it, but still no change, it has continued for five days, what should i do now.

    I did Fetch as Google for robots.txt and other pages many time, but it show Unreachable robots.txt, the error happened on January 16 to 17, the server of this web is in China, it always normal before January 16. We have another web with the same IP, it's ok now, but it's a Chinese web, and i was be told that Google has make it's Chinese server return to USA on January 17, may be it's the problem? The googlebot from USA can't crawl my web? Before that day my web crawled by googlebot from Beijing.
    Last edited by sirius; 02-20-2014 at 02:40 PM. Reason: Site linking not permitted.
      0 Not allowed!

  2. #2

    Please check firewall logs to see if Google's IPs are blocked, You can also test it by temporarily disabling firewall and check 'Fetch as Google'.

    This can happen also due to DNS issues. Please check your .htaccess too.
      0 Not allowed!

  3. #3
    Join Date
    Feb 2006
    Also recently the chinese firewall suffered serious issues and caused downtime, maybe this affected crawling?
      0 Not allowed!

  4. #4
    Join Date
    Jul 2012
    Check if your site is penalized
      0 Not allowed!

  5. #5
    check this query in google site:domain-name if you are able to see your pages than don't worry your site will be crawled and it is happening that some time google does not show the crawled status.
      0 Not allowed!

  6. #6
    Kindly check the crawling rate in webmaster tool,specially the last indexed date. Also check the DNS error and manual action section. I hope this will work.
      0 Not allowed!

  7. #7
    Another reason why Google is slow in spidering your site might be that your server is slow. Google actually indicates this to you.
      0 Not allowed!

  8. #8
    Perhaps you got penalty from google? If not, try to check and tidy up your site's internal linking because it's related to how fast google will recognize your site and crawl it. Google don't want to crawl a messy and incoherent website or if it did, very rarely and slowly.
      0 Not allowed!

  9. #9
    Join Date
    Mar 2014
    There will be many reasons for your issue which you need to check one by one and get confirmed that everything is ok on your side for Google to crawl your website. Please check the possible errors list i stated below

    1. No fresh content updated on pages for longer period
    2. Google might of penalized your website
    3. Reduced crawling rate in webmaster tools
    4. Your web hosting provider might have some firewalls enabled on the server which might be blocking google crawling your website
    5. Non responsive website/ too slow website

    Please do check with the other website when it was last cached by google, with that you will get some information on your IP problems also.
      0 Not allowed!

  10. #10
    First of check webmaster tools after that you can do social bookmarking with unique description.
      0 Not allowed!

  11. #11
    Join Date
    Mar 2014

    google stops crawling site help

    Check the source code of your home page and see what pages google stopped crawling by finding this (<meta name="robots" content="noindex,follow" />) code in your page source. If you are not getting this code then fetching your site fetch as google in webmaster will help you add your site again in google.
      0 Not allowed!

  12. #12
    Info from Support Google
    To see which URLs Google has been blocked from crawling, visit the Blocked URLs page of the Crawl section of Webmaster Tools.
      0 Not allowed!

Similar Threads

  1. Replies: 8
    Last Post: 04-21-2012, 09:53 AM
  2. Google crawl very fast
    By hellind2 in forum Web Hosting Lounge
    Replies: 15
    Last Post: 04-05-2012, 05:47 AM
  3. let google crawl your contents twice
    By Peter-SexyWing in forum SEO / SEM Discussions
    Replies: 6
    Last Post: 04-14-2010, 03:10 AM
  4. Google Crawl
    By HOD-Jardin in forum SEO / SEM Discussions
    Replies: 17
    Last Post: 09-17-2009, 08:01 AM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts