Results 1 to 13 of 13
  1. #1

    Exclamation how can i prevent form crawling my CentOS server

    hi all...

    i have VPS server with CentOS and i have one important question,i know crowling serves is a good way for hacker to get data on server,now i want to know how can i prevent from server crawling with crawler programms?
    tanxxxx

  2. #2
    Join Date
    May 2010
    Location
    Dendermonde, Belgium
    Posts
    147
    You can always block some by adding them to your robots.txt file in the root of your website dir.
    Take a closer look at ours: http://1ago.be/robots.txt - It's a good start to have those added.
    1Ago - Webhosting, Virtual Private Servers (VPS) & Dedicated Servers.
    Belgian company with servers located in The Netherlands!
    Managed services as an option.

  3. #3
    tanx man and how can i add "Acunetix Security Scanner" To your Robot LIST?

  4. #4
    Join Date
    May 2010
    Location
    Dendermonde, Belgium
    Posts
    147
    I'm guessing that's the User-Agent you mean.
    Just add a new line with this:

    User-agent: Acunetix Security Scanner
    Disallow: /
    1Ago - Webhosting, Virtual Private Servers (VPS) & Dedicated Servers.
    Belgian company with servers located in The Netherlands!
    Managed services as an option.

  5. #5
    tanxxxxx man this forum is best one in the world

  6. #6
    Join Date
    Aug 2010
    Location
    Brussels, Belgium
    Posts
    162
    Hi,

    I cannot agree. Forget about robots.txt here. robots.txt can be used with "polite" robots, but it is useless against ill-intended robots.

    This is from Acunetix documentation :
    Retrieve & process robots.txt – Select this option to have Acunetix WVS look for a robots.txt file and follow all the links in it.
    So the Acunetix robots can even use entries from robots.txt to discover vulnerabilities.

    If you know the user agent or the IP address they use, you can exclude it with a .htaccess file.

    Jean-Luc

  7. #7
    sorry bud how can i prevent form crawling? iwant any body cann't crawl my server contants! how can i do that with .htaccess i don't know much about .htaccess i know we can block ip or directory with this file, but i want to stop crowling!

    tanx again

  8. #8
    sorry bud how can i prevent form crawling? iwant any body cann't crawl my server contants! how can i do that with .htaccess i don't know much about .htaccess i know we can block ip or directory with this file, but i want to stop crowling!

    tanx again

  9. #9
    Join Date
    Dec 2007
    Location
    Indiana, USA
    Posts
    19,196
    If you don't want the content crawled, don't make it publicly available... I don't quite understand your question beyond that.
    Michael Denney - MDDHosting.com - Proudly hosting more than 37,700 websites since 2007.
    Ultra-Fast Cloud Shared and Pay-By-Use Reseller Hosting Powered by LiteSpeed!
    cPanel • Free SSL • 100% Uptime SLA • 24/7 Support
    Class-leading support that responds in minutes, not days.

  10. #10
    Join Date
    Aug 2010
    Location
    Brussels, Belgium
    Posts
    162
    If you are hosting public web sites, there is no way to completely prevent people or robots from crawling your sites. There are various ways to limit abuse, but it is a very broad subject that cannot just be covered in a few lines here.

    Maybe you can give more details about your environment.

    If all this sounds complicated to you, you might consider using a managed VPS service.

    Jean-Luc

  11. #11
    Join Date
    Mar 2009
    Posts
    3,816
    Use a snort-inline rule to just block the IP on the first attempt.

  12. #12
    The crawlers is no way to blocked. once you open your website to the world , the crawlers had the chance to crawl your website.

  13. #13

    Thumbs up

    Quote Originally Posted by quantumphysics View Post
    Use a snort-inline rule to just block the IP on the first attempt.
    hi man , how can i done this (snort-inline ) rule ? ishould set this rule in CSF Firewall?Plz explain...

    tanx

Similar Threads

  1. How to prevent server down?
    By sprakas4 in forum Running a Web Hosting Business
    Replies: 10
    Last Post: 02-11-2010, 03:17 PM
  2. Prevent other sites from leeching my form?
    By kohkindachi in forum Programming Discussion
    Replies: 4
    Last Post: 06-16-2009, 12:12 AM
  3. how to prevent *nobody* to move in server ?
    By boxer in forum Hosting Security and Technology
    Replies: 7
    Last Post: 06-15-2009, 08:12 PM
  4. Server Attached - How to prevent ??
    By rushik in forum Dedicated Server
    Replies: 1
    Last Post: 05-28-2005, 01:45 PM
  5. Please help! server crawling to a halt
    By ezclassified in forum Hosting Security and Technology
    Replies: 6
    Last Post: 10-02-2004, 12:04 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •