Results 1 to 4 of 4
  1. #1
    Join Date
    Feb 2006

    Wordpress blogs being crawled to death

    A very very common issue (affecting about 1 in 50 users) is that some customers use plugins for Wordpress that load remote content into their own blog. This creates about 20-30.000 pages. Now the issue is that these blogs are then being crawled to death by search engine bots. Each access is one PHP and MySQL execution and this happens 20-30.000 times within a few minutes.

    Does anyone have an idea how to handle this as a host ? To just suspend the site seems kind of unfair. Afterall the customer has done nothing illegal. And he has not overloaded the server either. Rather the crawlers are doing a ****** job by massively overloading the server. Is there some way to limit the crawler's access rate or something ?

  2. #2
    Join Date
    Nov 2005
    Advise your client to add wp-supercache plugin. This would create html pages for all these posts. As these pages would never change, this would be a perfect solution to reduce the cpu & mysql usage.
    Mellowhost - Providing High Quality Web Hosting Services since 2007
    SSD Cpanel Shared, SSD OpenVZ & KVM VPS Hosting
    A Hosting Provider with Complete SSD VPS & Shared Hosting.

  3. #3
    Join Date
    Feb 2006
    Awesome ! Thanks a lot for the help !

  4. #4
    Join Date
    Nov 2004
    A sitemap plugin might help too, as Google could check that rather than manually crawling ...

Similar Threads

  1. 2 wordpress blogs and 1 database - How?
    By bambinou in forum Programming Discussion
    Replies: 19
    Last Post: 12-01-2010, 05:52 PM
  2. vps for wordpress blogs
    By LawrenceV in forum VPS Hosting
    Replies: 12
    Last Post: 04-21-2008, 06:07 PM
  3. Advise me on server for wordpress blogs
    By myself1982 in forum Dedicated Server
    Replies: 7
    Last Post: 10-14-2007, 05:41 AM
  4. I need hosting for multiple wordpress blogs
    By Warlock77 in forum Web Hosting
    Replies: 9
    Last Post: 04-24-2006, 09:06 PM
  5. Blogs - Wordpress
    By SPLForums in forum Web Hosting
    Replies: 5
    Last Post: 03-02-2005, 03:55 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts