Results 1 to 5 of 5
  1. #1
    Join Date
    Sep 2011
    Posts
    36

    Crawled by 38.99.82.236?

    My website was crawled by a bot from this ip: 38.99.82.236. WHOIS here.
    Looks like the ip belongs to PSINet, Inc. which was then acquired by Cogent Communications, which is a legitimate telecommunications company. See their wiki here and here. Their bot(s) obviously ignored my robots.txt

    But I don't understand why they would crawl my website to begin with? My DNS is hosted with HE (using their free service). The wiki page says "Cogent has yet to agree on peering with one of the biggest IPv6 connectivity providers, Hurricane Electric." Could this have anything to do with it?

    Btw, how would I deny the entire range 38.0.0.0 - 38.255.255.255 that belongs to them in nginx?
    Last edited by hitman047; 10-20-2011 at 11:40 PM.

  2. #2
    Join Date
    Mar 2007
    Location
    Phoenix, AZ, United State
    Posts
    1,525
    Cogent Communications would be the result you'd get from many many many thousands of servers from many different organizations.
    drew@slicie.com - Vertical Scaling Servers
    30 Minute Backups - Pay for what you use

  3. #3
    To deny the range in nginx, just add:
    Code:
    deny 38.0.0.0/8;

  4. #4
    Join Date
    May 2008
    Location
    Citrus Heights, CA
    Posts
    1,887
    Cogent leases IPs from this block; I wouldn't block the whole range. Just block 38.99.82.* if you must.
    iWebFusion.Net - Shared / Reseller / VPS / Bare Metal / Colocation / IP Transit / Networking
    *Simply Hosting - Wholly owned networks, in-house staff, legions of fans!

  5. #5
    Join Date
    Apr 2009
    Location
    Mule Barn
    Posts
    1,571
    They've been around for a long time. I wouldn't waste to much time trying to stop them unless they were hitting you with piles of hits a day.

Similar Threads

  1. Wordpress blogs being crawled to death
    By glace in forum Hosting Security and Technology
    Replies: 3
    Last Post: 06-10-2011, 07:54 AM
  2. Why Do All My Blog Posts Get Crawled So Fast By Google?
    By growpublicity in forum SEO / SEM Discussions
    Replies: 8
    Last Post: 04-14-2011, 08:59 AM
  3. Australian Web Host For Sale - 236 customers / revenue $19,789
    By auhost-sale07 in forum Other Web Hosting Related Offers
    Replies: 7
    Last Post: 02-25-2007, 07:41 PM
  4. Happywebmasters.com Developed Forum for sale $236 profit a month
    By tomcromp in forum Other Offers & Requests
    Replies: 20
    Last Post: 11-08-2005, 06:55 PM
  5. Verified.net, Landed.com, Crawled.com
    By checknme in forum Domain Name Offers
    Replies: 3
    Last Post: 11-16-2003, 12:47 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •