Hi,

I have previous post here if you want to know more about me case: click here (http://www.webhostingtalk.com/showthread.php?threadid=7137)

My site is reaching in this month 9 million hits (9777631). I have two sites with my hosting company one of them is very normal, but the other one is crazy and I don't know how to deal with it and solve the problem.
The problem with second site is it’s making a lot of hits; it is now 10 days and the second site is down because it exceeds its bandwidth limit.

I tried to block some IPs! It works for one day then everything jumps up again. The site is making (407401) avg. hits a day. The most active part in the site is a discussion board like this one, with avg. 25-30 user and 30 posts a day.

I'm not sure if this is a kind of attack that causing this problem? If yes what it is call, and how to stop it?

One more thing, In the admin control panel there is an IP made 1 visit with 100986 Hits!! Is this possible? Another one made 1 visit with 64690 Hits. There are many like this, may be they are many people share same IP or not, I'm not sure!!

Please advice me what to do, I’m on share server, and I have a BIG doubt that these are true numbers or visitors.. Thanks

What does the IP address that had so many hits resolve as? It could well have been a search engine on it's spider run, it will follow each link on each page. This includes the edit, reply, profile and email links unless you have specified otherwise in your meta tags.

I had this problem with our gaming site, all of a sudden Google came along and pinched about 250mb of bandwidth in one day.

Then I installed the meta tags and all has been fine since :)..

Rich

This is one of the IPs (62.252.64.5) making a huge bandwidth usage. The site is closed right now becuase I exceeded the bandwidth limit but the above IP for example is still making #1 of hits!!

What is the meta tag that you used? did you use it in each single page in your site?

I used [meta name="robots" content="noindex,nofollow"] in my meta tags, and only in the forums(incorporate it in the header). You'll need to change the < to [ and the > to ].

The IP address 62.252.64.5 resolves as "inktomi2-lit.server.ntl.com" according to this (http://www.amnesi.com/hostinfo/ipinfo.jhtml?ip1=62&ip2=252&ip3=64&ip4=5&Search=Lookup+Name&wholeIp=62.252.64.5). NTL are a communications company here in the UK, but I'm sure I heard that inktomi were a search engine of sorts? The free listings at GoTo.com are via inktomi I think.

I don't understand why it would come up as an NTL address. I'd put the code into the forums and see what happens. If they come again, I'd contact abuse@ntl.com and explain whats happening.

Hope this helps...

Rich

[Edited by Inspa.net on 03-25-2001 at 08:41 PM]

Inspa what is the meta tag code that you used to stop the spider search?

Sorry it didn't show up, I've edited the post now :).

I really should read my posts after I make them ;).

Rich

FYI, inktomi2-lut.server.ntl is a caching proxy server. All port 80 (web) requests by ntl customers are transparently redirected to one of the local proxy servers. In this case, the inktomi2-lut indicates that this is probably the second of a cluster of proxy servers, and this is based in the lutterworth region in the UK.

(Referring to Vincent Paglione's post in the other thread you mentioned) It's doubtful that your web host would want to block a proxy server on a shared server, since that may impact other websites on that server.

This might possibly be some kind of an attack in order to make you use up a lot of bandwidth, but... I don't know. The proxy server may be aggregating traffic from hundreds of people visiting your site from lutterworth, or perhaps it's just someone with a cablemodem grabbing everything :/

Perhaps you should just post a message on the bulletin board asking everyone using ntl to make a post there, just to see how many users you are getting from there?

If you decide that you don't want anyone using ntl in the uk to have access to your BB, you could do as Vincent suggested in the other post. I'd suggest using "deny from .ntl.com" or "deny from .server.ntl.com"

Or perhaps you could even run your BB on a different port than port 80, so that the traffic won't automatically go via the proxy.

P.S. the proxy servers aren't reducing your bandwidth figures much because bulletin boards are highly dynamic, and probably return a Pragma: no-cache or something similar in order to prevent users from seeing a stale page. It should at least help reduce hits on the few graphic elements that are static.

Isn't this what the robots.txt file is used for?

If you look at where this post came from (the IP listed below) you can see I am behind one of those cache servers the Brimingham one. They are usually ok but sometimes when 1 crashes and every one starts using the other things go really sloooooooow.

NTL has lots of customers by the way.

I reached to 10 million hits (10281284) untill this date :(

I don't have many people from UK that access my site. My host company reply and said they cannot block an IP on share server, and they advice me to use anti-leech but I don't think anti-leech can do anything in my case because it is for differnet use.
http://www.antileech.com
http://www.antileech.net

How I can run BB on a different port than 80?
At this time I will still use Vincent Paglione's trick

Are you running on a Linux server? If so, you can always block an IP using the .htaccess file.

Rich

Originally posted by astra4
Isn't this what the robots.txt file is used for?

robots.txt is meant to tell web spiders and other automatons what they may or may not retrieve. It only works with well behaved spiders which actually abide by what the file says, since it doesn't enforce anything on its own.

Ordinary web browsers (such as the ones you and me use to visit a web forum) are unlikely to care about them. If someone were indeed trying to attack his site by generating a lot of traffic and tying up CPU, he is unlikely to care about robots.txt too :)

Guys I reached 11 million hits (11191452), this is very bad, I don't know what to do!! :(

I tried to block some IPs through .htaccess but it seems it is not working. This IP (62.252.64.5) still making #1 hits (385745). Look at this marseille-1-a7-51-41.dial.proxad.net (1 visit , 81841 hits) & 216-164-194-251.s632.tnt5.atn.pa.dialup.rcn.com (1 visit , 47399 hits) can 1 visit to a site make all these hits? There many like these.
My host want me to move to dedicated server. I think with this situation even 100 Gig bandwidth will not be enough for me. I just don't know what to do! :(

The other site I have with same host is just fine and perfect, and it has more stuff on the site than this one! Do you think there are files or wrong setting causing this?

Without a URL to your site, you are making us "guess" what the problem could be. Also, hits don't mean much. You should refer to your Bandwidth in MB as that makes more sense.

With that in mind let me give you an example:

I offered a Free program which ended up, with the link for downloading, being put on other people's Web sites and lots of Bulletin Boards - some of them are very popular. Soon I was using 2-3 hundred MB of Bandwidth per month - for that "one" program - with virtually no new visitors to my site. Not good for me.

What I did was setup an Anti-leech program and made it so people "have to" visit the site, in order to obtain the program. So if you have something on your site -- and others have found it popular/beneficial, you could be in the same position. That's why the recommendation for using an Anti-leech script.

You should be looking in your Stats (or log files) to see "which" file or program, is getting the most attention. Maybe you too, have a downloadable program or some graphics, that other sites have linked to? Maybe your site is fantastic and lots of people are using "Site Grabber" programs, to grab everything.

Maybe, maybe, maybe...

Hard to give you good advice, with so little information to go on.

Hi, I would prefer to use robots.txt rather than meta tag... any hint what to put in the robots.txt?

TIA

Hello,

Check out this link for info on robots.txt

http://www.searchtools.com/robots/robots-txt.html

Regards,

Mike

I used robot.txt
-------------
User-agent: *
Disallow: /
-------------
and I changed some foulders names, the hits reduced a big time from (494316) to (3777) - the difference is 490539 - this is a static of one day. I'm still not sure what was the reason since I used many trick as pepole post here, I think I have to monitor the site for next days.
What I'm doing now is the site is closed for public, and I will try to get things back one after another and see what changes will happen for the hits

Originally posted by jh
Hi, I would prefer to use robots.txt rather than meta tag... any hint what to put in the robots.txt?

TIA

Take a look at www.rietta.com/robogen. (http://www.rietta.com/robogen) RoboGen is a program that my company released two years ago that makes it easier to make robots.txt files. You can download the shareware or the freeware version to help make your file.

Best regards,
Frank Rietta