Results 1 to 10 of 10
  1. #1
    Join Date
    Mar 2004
    Posts
    39
    would you tell me for any search engine this crwler belong and why it has a lot consumption of bandwidth :
    Unknown robot (identified by 'crawl') 32685 608.79 MB 11 Jun 2005 - 00:42

    600 mbs/three days

  2. #2
    Join Date
    Oct 2003
    Location
    West Yorkshire, UK
    Posts
    2,813
    wisam74us,

    I'm not sure what search engine, if any, that bot belongs to. If that bandwidth usage is excessive, on average, for a bot at your Website, then I would consider attempting to block it.

    It may well be something that is totally reasonable, but to be on the safe side, blocking it may be the answer.

    I have heard about cases where a bot became, 'trapped', in a sense, within a Website. This often occurs in forums, and so the bot crawls and crawls until it finds a way out, in effect, eating up a lot of bandwidth.
    - Jamie Harrop

  3. #3
    Join Date
    Jul 2003
    Location
    Castle Pines, CO
    Posts
    7,189
    Do you have access to the raw logs with the IP address?

  4. #4
    Whoa that sucks, hope you find out.

  5. #5
    the same on my site

    2 ips

    148.244.150.58 [mexico ip]
    207.248.240.119 [caribean ip]


    for last 2-3 days i have 1.68gb from first ip and 1.3gb from the second ip.

    do u know any caribean or mexic search engines?

    what to do? block them
    Your Health Encyclopedia
    Medical and health consumer information resources containing comprehensive and unbiased information in patient-friendly language

  6. #6
    Join Date
    Sep 2003
    Posts
    1,211
    Might not be a bot, could be some form of attack?

  7. #7
    but i check the same ip's in my logs for last 2-3 month

    hmm .. btw i check a lot of errors like this

    public_html/MSOffice/cltreq.asp
    public_html/_vti_bin/owssvr.dll

    but from other ip's

    u may be right. but with a such huge traffic ?
    Your Health Encyclopedia
    Medical and health consumer information resources containing comprehensive and unbiased information in patient-friendly language

  8. #8
    Definately block before they use all of your bandwidth

  9. #9
    k .. i do it

    thx
    Your Health Encyclopedia
    Medical and health consumer information resources containing comprehensive and unbiased information in patient-friendly language

  10. #10
    look what i find when i download Raw Access Logs

    both ips

    207.248.240.119 - - [31/May/2005:17:23:50 -0400] "GET /weblog.php?id=C0_29_1 HTTP/1.1" 200 20608 "http://www.progressiveupdate.[deleted by me]/party-poker.html" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; .NET CLR 1.0.3705)"

    148.244.150.58 - - [13/Jun/2005:12:53:10 -0400] "GET /more.php?id=14749_0_1_0_C HTTP/1.1" 200 70747 "http://online-casino-gambling.casino-ppp.{delete by me}" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; N_o_k_i_a)"

    F***

    how can i report such scam attacks? is there any way (like report spam), so to stop this guys.
    Your Health Encyclopedia
    Medical and health consumer information resources containing comprehensive and unbiased information in patient-friendly language

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •