Results 1 to 13 of 13
-
01-08-2011, 11:47 AM #1Newbie
- Join Date
- Jan 2011
- Posts
- 9
how can i prevent form crawling my CentOS server
hi all...
i have VPS server with CentOS and i have one important question,i know crowling serves is a good way for hacker to get data on server,now i want to know how can i prevent from server crawling with crawler programms?
tanxxxx
-
01-08-2011, 11:53 AM #2WHT Addict
- Join Date
- May 2010
- Location
- Dendermonde, Belgium
- Posts
- 147
You can always block some by adding them to your robots.txt file in the root of your website dir.
Take a closer look at ours: http://1ago.be/robots.txt - It's a good start to have those added.1Ago - Webhosting, Virtual Private Servers (VPS) & Dedicated Servers.
Belgian company with servers located in The Netherlands!
Managed services as an option.
-
01-08-2011, 12:01 PM #3Newbie
- Join Date
- Jan 2011
- Posts
- 9
tanx man and how can i add "Acunetix Security Scanner" To your Robot LIST?
-
01-08-2011, 12:03 PM #4WHT Addict
- Join Date
- May 2010
- Location
- Dendermonde, Belgium
- Posts
- 147
I'm guessing that's the User-Agent you mean.
Just add a new line with this:
User-agent: Acunetix Security Scanner
Disallow: /1Ago - Webhosting, Virtual Private Servers (VPS) & Dedicated Servers.
Belgian company with servers located in The Netherlands!
Managed services as an option.
-
01-08-2011, 12:11 PM #5Newbie
- Join Date
- Jan 2011
- Posts
- 9
tanxxxxx man this forum is best one in the world
-
01-08-2011, 04:22 PM #6WHT Addict
- Join Date
- Aug 2010
- Location
- Brussels, Belgium
- Posts
- 162
Hi,
I cannot agree. Forget about robots.txt here. robots.txt can be used with "polite" robots, but it is useless against ill-intended robots.
This is from Acunetix documentation :Retrieve & process robots.txt – Select this option to have Acunetix WVS look for a robots.txt file and follow all the links in it.
If you know the user agent or the IP address they use, you can exclude it with a .htaccess file.
Jean-Luc
-
01-08-2011, 04:30 PM #7Newbie
- Join Date
- Jan 2011
- Posts
- 9
sorry bud how can i prevent form crawling? iwant any body cann't crawl my server contants! how can i do that with .htaccess i don't know much about .htaccess i know we can block ip or directory with this file, but i want to stop crowling!
tanx again
-
01-08-2011, 05:01 PM #8Newbie
- Join Date
- Jan 2011
- Posts
- 9
sorry bud how can i prevent form crawling? iwant any body cann't crawl my server contants! how can i do that with .htaccess i don't know much about .htaccess i know we can block ip or directory with this file, but i want to stop crowling!
tanx again
-
01-08-2011, 05:03 PM #9Web Hosting Industry Expert
- Join Date
- Dec 2007
- Location
- Indiana, USA
- Posts
- 19,196
If you don't want the content crawled, don't make it publicly available... I don't quite understand your question beyond that.
█ Michael Denney - MDDHosting.com - Proudly hosting more than 37,700 websites since 2007.
█ Ultra-Fast Cloud Shared and Pay-By-Use Reseller Hosting Powered by LiteSpeed!
█ cPanel • Free SSL • 100% Uptime SLA • 24/7 Support
█ Class-leading support that responds in minutes, not days.
-
01-08-2011, 05:04 PM #10WHT Addict
- Join Date
- Aug 2010
- Location
- Brussels, Belgium
- Posts
- 162
If you are hosting public web sites, there is no way to completely prevent people or robots from crawling your sites. There are various ways to limit abuse, but it is a very broad subject that cannot just be covered in a few lines here.
Maybe you can give more details about your environment.
If all this sounds complicated to you, you might consider using a managed VPS service.
Jean-Luc
-
01-08-2011, 08:39 PM #11Web Hosting Master
- Join Date
- Mar 2009
- Posts
- 3,816
Use a snort-inline rule to just block the IP on the first attempt.
-
01-08-2011, 08:53 PM #12Aspiring Evangelist
- Join Date
- Jan 2010
- Posts
- 389
The crawlers is no way to blocked. once you open your website to the world , the crawlers had the chance to crawl your website.
-
01-09-2011, 05:22 AM #13Newbie
- Join Date
- Jan 2011
- Posts
- 9
Similar Threads
-
How to prevent server down?
By sprakas4 in forum Running a Web Hosting BusinessReplies: 10Last Post: 02-11-2010, 03:17 PM -
Prevent other sites from leeching my form?
By kohkindachi in forum Programming DiscussionReplies: 4Last Post: 06-16-2009, 12:12 AM -
how to prevent *nobody* to move in server ?
By boxer in forum Hosting Security and TechnologyReplies: 7Last Post: 06-15-2009, 08:12 PM -
Server Attached - How to prevent ??
By rushik in forum Dedicated ServerReplies: 1Last Post: 05-28-2005, 01:45 PM -
Please help! server crawling to a halt
By ezclassified in forum Hosting Security and TechnologyReplies: 6Last Post: 10-02-2004, 12:04 PM