Results 1 to 5 of 5
Thread: Crawled by 38.99.82.236?
-
10-20-2011, 11:35 PM #1Junior Guru Wannabe
- Join Date
- Sep 2011
- Posts
- 36
Crawled by 38.99.82.236?
My website was crawled by a bot from this ip: 38.99.82.236. WHOIS here.
Looks like the ip belongs to PSINet, Inc. which was then acquired by Cogent Communications, which is a legitimate telecommunications company. See their wiki here and here. Their bot(s) obviously ignored my robots.txt
But I don't understand why they would crawl my website to begin with? My DNS is hosted with HE (using their free service). The wiki page says "Cogent has yet to agree on peering with one of the biggest IPv6 connectivity providers, Hurricane Electric." Could this have anything to do with it?
Btw, how would I deny the entire range 38.0.0.0 - 38.255.255.255 that belongs to them in nginx?Last edited by hitman047; 10-20-2011 at 11:40 PM.
-
10-20-2011, 11:40 PM #2Web Hosting Master
- Join Date
- Mar 2007
- Location
- Phoenix, AZ, United State
- Posts
- 1,525
Cogent Communications would be the result you'd get from many many many thousands of servers from many different organizations.
drew@slicie.com - Vertical Scaling Servers
30 Minute Backups - Pay for what you use
-
10-21-2011, 12:21 AM #3Newbie
- Join Date
- Oct 2011
- Posts
- 13
To deny the range in nginx, just add:
Code:deny 38.0.0.0/8;
-
10-21-2011, 12:49 AM #4Rebooting is a hack, not a fix
- Join Date
- May 2008
- Location
- Citrus Heights, CA
- Posts
- 1,887
Cogent leases IPs from this block; I wouldn't block the whole range. Just block 38.99.82.* if you must.
iWebFusion.Net - Shared / Reseller / VPS / Bare Metal / Colocation / IP Transit / Networking
*Simply Hosting - Wholly owned networks, in-house staff, legions of fans!
-
10-21-2011, 12:53 AM #5Web Hosting Master
- Join Date
- Apr 2009
- Location
- Mule Barn
- Posts
- 1,571
They've been around for a long time. I wouldn't waste to much time trying to stop them unless they were hitting you with piles of hits a day.
Similar Threads
-
Wordpress blogs being crawled to death
By glace in forum Hosting Security and TechnologyReplies: 3Last Post: 06-10-2011, 07:54 AM -
Why Do All My Blog Posts Get Crawled So Fast By Google?
By growpublicity in forum SEO / SEM DiscussionsReplies: 8Last Post: 04-14-2011, 08:59 AM -
Australian Web Host For Sale - 236 customers / revenue $19,789
By auhost-sale07 in forum Other Web Hosting Related OffersReplies: 7Last Post: 02-25-2007, 07:41 PM -
Happywebmasters.com Developed Forum for sale $236 profit a month
By tomcromp in forum Other Offers & RequestsReplies: 20Last Post: 11-08-2005, 06:55 PM -
Verified.net, Landed.com, Crawled.com
By checknme in forum Domain Name OffersReplies: 3Last Post: 11-16-2003, 12:47 PM