Results 1 to 16 of 16
  1. #1
    Join Date
    Aug 2013
    Location
    Bhubaneswar, India
    Posts
    504

    Bots loading up the server ?

    My websites go down a lot of times in a day . My host said I'm reaching max clients most of the time.

    I looked a bit into it and found some bots hitting 40-50k from the same IP to the websites .

    Contacted a friend and be said he had similar issue and he changed his PHP handler to DSO and mod_ruid2 .

    So what does he mean ?
    JetCloud Datacenter Pvt Ltd - Managed VPS / Dedicated Servers India
    AS134325 - Powered by JetSpot Networks
    Shared, Reseller, VPS, Dedicated Servers, Colocation
    Own Dedicated Network Backbone - Core (Juniper) Aggregation (Huawei) Access (Cisco)

  2. #2
    Join Date
    Feb 2013
    Posts
    602
    It is quite complicated handling bots yourself, you would need a Layer 7 Analyzer for this, or try the hard way, with user agent filtering, ip blocking (when too much connections are made), well quite a lot of hand work honestly, not to mention that if the load is too high, it will not solve the issue.
    Mark
    IT / software / infrastructure / security projectist.
    Specializing in datacenter environments and threat mitigation.
    Your success, is our success !

  3. If you have access to cpanel admin, you can try to load csf from config server, it's an easy way to mitigate this kind of issue.
    Hostabulous | cPanel (Linux) & Plesk (Windows) Hosting KVM VPS R1Soft backups | Proudly Canadian
    Cloudflare LiteSpeed Cloudlinux Remote backups Anti-Spam Web App Firewall Canada/US/Germany

  4. #4
    Join Date
    Aug 2013
    Location
    Bhubaneswar, India
    Posts
    504

    Bots loading up the server ?

    I asked the host to find all the bots accessing and block them all except google and bing bot.

    Is that okay ?
    JetCloud Datacenter Pvt Ltd - Managed VPS / Dedicated Servers India
    AS134325 - Powered by JetSpot Networks
    Shared, Reseller, VPS, Dedicated Servers, Colocation
    Own Dedicated Network Backbone - Core (Juniper) Aggregation (Huawei) Access (Cisco)

  5. #5
    Join Date
    May 2007
    Posts
    1,979

    Re: Bots loading up the server ?

    Nope. Why block yandex and majesticseo for example?

  6. #6
    Join Date
    Oct 2013
    Posts
    45
    Quote Originally Posted by soamz View Post
    I asked the host to find all the bots accessing and block them all except google and bing bot.

    Is that okay ?
    This is not the correct thing to do, someone can easily bypass the block only changing the user-agent.
    HyperFilter DDoS Protection Solutions
    Specializing in DDoS Protected Hosting Services such as:
    Dedicated Servers - Colocation - WebHosting - Domain Registration - Remote Protection Services
    Exclusive management panel for attack reporting available for all services.

  7. #7
    Join Date
    Aug 2013
    Location
    Bhubaneswar, India
    Posts
    504

    Bots loading up the server ?

    Okay so unblock those two
    JetCloud Datacenter Pvt Ltd - Managed VPS / Dedicated Servers India
    AS134325 - Powered by JetSpot Networks
    Shared, Reseller, VPS, Dedicated Servers, Colocation
    Own Dedicated Network Backbone - Core (Juniper) Aggregation (Huawei) Access (Cisco)

  8. #8
    Join Date
    May 2007
    Posts
    1,979

    Re: Bots loading up the server ?

    Indeed. It's fairly simple to code

  9. #9
    Join Date
    Mar 2013
    Posts
    1,294
    40-50k? How many websites are we talking about?

  10. #10
    Join Date
    Nov 2004
    Location
    Dallas
    Posts
    739
    Have you considered cache alternatives such as CloudFlare?

  11. #11
    Rather than doing shared hosting, you may want to move up to a VPS, and then take control over your server and hosting environment. It really won't cost much more than you are already spending, but you will have access to everything and you can let the bots hit it all day long without worrying about it.
    Managed Service Provider - www.OpticIP.com
    Public & Private Cloud
    Solutions | SSD SANs | High IOP's | CDN Solutions
    Phoenix/Chandler AZ Colocation | 48U Cabinets | Data Halls | TIA-942 Tier 4 Facility

  12. #12
    Join Date
    Jan 2010
    Location
    USA
    Posts
    2,148
    Quote Originally Posted by soamz View Post
    I asked the host to find all the bots accessing and block them all except google and bing bot.

    Is that okay ?
    Bots are not always honest about who they are. I have seen logs of bots claiming to be BingBot, but where actually just hackers running brute force login attacks against WordPress's wp-login.php page. Look at the raw logs. Are the bots hitting files frequently that seems weird for a search engine indexing bot? For example, search engines don't need to index your login pages thousands of times per minute. If you see weird activity like that, it probably is not a real search engine bot.
    No Support Linux Hosting Bargain cPanel Hosting Experts Only
    We IGNORE the support questions, and pass the SAVINGS on to YOU!
    We also ignore questions about VPS Hosting

  13. Also get some proper mod_sec rules running
    Hostabulous | cPanel (Linux) & Plesk (Windows) Hosting KVM VPS R1Soft backups | Proudly Canadian
    Cloudflare LiteSpeed Cloudlinux Remote backups Anti-Spam Web App Firewall Canada/US/Germany

  14. #14
    Join Date
    Mar 2013
    Location
    North and South America
    Posts
    166
    I use mod_security with ConfigServer's ModSecurity Control based bot blocking. For the honest, but still unworthy bots I use this: http://puntapirata.com/On-House-ModSec-Rules.php. Then to cut out the dishonest liars I use custom mod_security rules (modified from gotroot) with apache http.conf configured with "HostnameLookups Double" against an IP white list for good bots. This works wonderfully.

    On some servers we have the puntapirata bad bot lists (we have 1200+ bots listed) broken into a few groups (separate rules). This way customers can opt out of banning any bot (or a group).

    Server load is much less and it protects our clients from unscrupulous harvesting.

  15. #15
    Join Date
    Mar 2013
    Location
    North and South America
    Posts
    166
    My sincere apologies! I gave you my recommendations and forgot to answer your direct questions.

    My websites go down a lot of times in a day . My host said I'm reaching max clients most of the time.

    I looked a bit into it and found some bots hitting 40-50k from the same IP to the websites .

    Contacted a friend and be said he had similar issue and he changed his PHP handler to DSO and mod_ruid2 .

    So what does he mean ?
    PHP can be loaded in a variety of ways. DSO is a particular php handler, you might be using or try SuPHP as well. mod_ruid2 is an Apache extension that differently treats ALL requests (except JS/Java) to run as the domain owner not apache user. Some nice security in that way. Unfortunately, mod_ruid2 is a big tradeoff with many apps/extensions/environmental settings so you can't run Tomcat, mod_security, MemCache, FastCGI, etc. if you use mod_ruid2. OK, people don't use Tomcat much anyway anymore. But mod_security as all others have chimed in is a good part of many solutions. Frankly, I would not know how to whip up a good recipe that way. I think that is why I forgot to answer your question. To me I don't see the route there.

  16. #16
    Join Date
    Dec 2011
    Location
    Germany
    Posts
    1,154
    Sounds like these are bad bots. You could either block them by using a good DDoS protection, or you can try to optimize your server so it can handle all the requests. A fast PHP handler such as mod_php or php-fpm can help in this case, just as the other tips already mentioned by other users.
    Inbound Marketing & real SEO for web hosting providers
    ✎ Get in touch with me: co<at>infinitnet.de

Similar Threads

  1. Complaints of BOTS on my server
    By Joels341 in forum Hosting Security and Technology
    Replies: 6
    Last Post: 03-03-2008, 10:53 PM
  2. 100 irc bots on a server
    By Jippi in forum Dedicated Server
    Replies: 13
    Last Post: 11-18-2004, 07:29 PM
  3. Setting up bots for a game server!
    By Moose in forum Employment / Job Offers
    Replies: 4
    Last Post: 04-29-2004, 08:01 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •