Results 1 to 22 of 22
  1. #1
    Join Date
    Jul 2009
    Posts
    40

    * Amazing - Bandwidth Thief Transfer - Magic Issue

    Hi All,

    I am undergoing "amazing" Bandwidth thief issue. My Server is using very high bandwidth from last 7 days -

    On Scanning server, i found following

    1) Roundcube mail script was compromised and used by some hacking group, so i deleted that directory all together
    2) I disabled Hotlinking all together
    3) I found "Live Chat" script also being compromised, and found movies (.dat files) in it amounting to 20 GB++ , i also deleted that directory
    4) I have checked for each and every file, with size more than 30 MB on server.
    5) I have also switched off FTP service (and there is no other user on server too)
    6) After all this, i even installed "mod_bandwidth" and limited bandwidth service to 10 KBPS across domain directory

    Then, i installed MRTG graphs to analyze the issue, and to my SURPRISE, nothing changed, it still used very high bandwidth, after few - few hours.

    http://img208.imageshack.us/img208/511/indexday.png

    ^^ the above image shows B/W goes upto 10 MBPS and thus breaks network.

    I am now tired and exhausted of this. Can someone here, please give an expert advice on this issue ?

    I hope some experience person will give an insight into this issue.

  2. #2
    Join Date
    Mar 2009
    Location
    /home/khunj
    Posts
    432
    Quote Originally Posted by Leaptopz View Post
    3) I found "Live Chat" script also being compromised, and found movies (.dat files) in it amounting to 20 GB++ , i also deleted that directory
    According to your previous post, you have an IRC shell script running. If you don't have any IRC server, then block port 6667 with iptables first :

    Code:
    iptables -I INPUT -p tcp --dport 6667 -j DROP
    Then kill both executable files that were in the stealth.tgz archive (run `ps aux | grep 'name_of_the_file'`) :
    -rwxr-xr-x 1 root root 13399 2005-07-05 18:38 stealth
    -rwxr-xr-x 1 root root 590481 2005-07-14 19:51 syslogd

    Be careful not to kill /sbin/syslogd !

  3. #3
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,571
    Wow, sounds like you really need mod_security.
    Fast Serv Networks, LLC | AS29889 | Fully Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  4. #4
    Join Date
    Mar 2008
    Posts
    1,717
    If there's any processes running as root that aren't supposed to be, just assume the security of the entire machine is tainted and backup + reinstall. It's not worth your time hunting rootkits, when all they have to do is leave one where you slip up once.

    If they haven't got root, continue the cleaning process... keep stepping on things as you find them, and also keep an eye out for c99 type shells. I believe there's mod_security rules that'll block stuff like this automagically - you might try that.
    I used to run the oldest commercial Mumble host.

  5. #5
    Join Date
    Jul 2009
    Posts
    40
    Quote Originally Posted by khunj View Post
    According to your previous post, you have an IRC shell script running. If you don't have any IRC server, then block port 6667 with iptables first :

    Code:
    iptables -I INPUT -p tcp --dport 6667 -j DROP
    Then kill both executable files that were in the stealth.tgz archive (run `ps aux | grep 'name_of_the_file'`) :
    -rwxr-xr-x 1 root root 13399 2005-07-05 18:38 stealth
    -rwxr-xr-x 1 root root 590481 2005-07-14 19:51 syslogd

    Be careful not to kill /sbin/syslogd !
    Thanks for replying, on your advice i have executed -

    Code:
    iptables -I INPUT -p tcp --dport 6667 -j DROP
    I did not found the file 'stealth.tgz' in my system

    Here is out put of my 'top' command -

    Code:
    top - 21:24:23 up  4:01,  1 user,  load average: 0.39, 0.27, 0.16
    Tasks: 117 total,   1 running, 116 sleeping,   0 stopped,   0 zombie
    Cpu(s):  2.7% us,  0.3% sy,  0.0% ni, 97.0% id,  0.0% wa,  0.0% hi,  0.0% si
    Mem:   3107900k total,   711844k used,  2396056k free,    54508k buffers
    Swap:  2096472k total,        0k used,  2096472k free,   438472k cached
    
      PID USER      PR  NI %CPU    TIME+  %MEM  VIRT  RES  SHR S COMMAND
    14366 apache    15   0  1.7   0:01.25  0.3 28756  10m 2612 S httpd
    15113 apache    15   0  0.3   0:00.55  0.3 28944  10m 2628 S httpd
        1 root      16   0  0.0   0:00.49  0.0  2508  548  468 S init
        2 root      34  19  0.0   0:00.00  0.0     0    0    0 S ksoftirqd/0
        3 root       5 -10  0.0   0:00.00  0.0     0    0    0 S events/0
        4 root       5 -10  0.0   0:00.01  0.0     0    0    0 S khelper
        5 root      15 -10  0.0   0:00.00  0.0     0    0    0 S kacpid
       27 root       5 -10  0.0   0:00.00  0.0     0    0    0 S kblockd/0
       28 root      15   0  0.0   0:00.00  0.0     0    0    0 S khubd
       45 root      20   0  0.0   0:00.00  0.0     0    0    0 S pdflush
       46 root      15   0  0.0   0:00.00  0.0     0    0    0 S pdflush
       47 root      25   0  0.0   0:00.00  0.0     0    0    0 S kswapd0
       48 root      12 -10  0.0   0:00.00  0.0     0    0    0 S aio/0
      194 root      25   0  0.0   0:00.00  0.0     0    0    0 S kseriod
      415 root       5 -10  0.0   0:00.00  0.0     0    0    0 S ata/0
      416 root       6 -10  0.0   0:00.00  0.0     0    0    0 S ata_aux
      418 root      15   0  0.0   0:00.00  0.0     0    0    0 S scsi_eh_0


    Quote Originally Posted by fwaggle View Post
    If there's any processes running as root that aren't supposed to be, just assume the security of the entire machine is tainted and backup + reinstall. It's not worth your time hunting rootkits, when all they have to do is leave one where you slip up once.

    If they haven't got root, continue the cleaning process... keep stepping on things as you find them, and also keep an eye out for c99 type shells. I believe there's mod_security rules that'll block stuff like this automagically - you might try that.
    ^^ i have pasted 'top' results above, please check.
    I can not reinstall entire machine, as it has GB's of data (specially mysql databases on it)

    I have changed 'root' password.

    Can you help me on how to find c99 type shells.

    In the meantime i will install mod_security, by reading help some where.

    Thanks for your precious post.

  6. #6
    Join Date
    Jul 2009
    Posts
    40

    Mod_security for Apache 1.3x installed

    Ok, finally i have also installed Mod_security for Apache 1.3x with following configuration setting in 'httpd.conf' file

    Code:
    <IfModule mod_security.c>
    SecFilterEngine On
    
    SecServerSignature "Apache"
    SecFilterCheckUnicodeEncoding Off
    SecAuditEngine RelevantOnly
    SecAuditLog logs/audit_log
    SecFilterScanPOST On
    
    SecFilterDefaultAction "deny,log,status:403"
    
    SecFilterSelective REQUEST_METHOD "^POST$" chain
    SecFilterSelective HTTP_Content-Length "^$"
    
    SecFilterSelective HTTP_Transfer-Encoding "!^$"
    
    SecFilterSelective ARG_PHPSESSID "!^[0-9a-z]*$"
    SecFilterSelective COOKIE_PHPSESSID "!^[0-9a-z]*$"
    SecFilter "../"
    
    SecFilter "viewtopic\.php\?" chain
    SecFilter "chr\(([0-9]{1,3})\)" "deny,log"
    
    SecFilterSelective THE_REQUEST "wget "
    SecFilterSelective THE_REQUEST "lynx "
    SecFilterSelective THE_REQUEST "scp "
    SecFilterSelective THE_REQUEST "ftp "
    SecFilterSelective THE_REQUEST "cvs "
    SecFilterSelective THE_REQUEST "rcp "
    SecFilterSelective THE_REQUEST "curl "
    SecFilterSelective THE_REQUEST "telnet "
    SecFilterSelective THE_REQUEST "ssh "
    SecFilterSelective THE_REQUEST "echo "
    SecFilterSelective THE_REQUEST "links -dump "
    SecFilterSelective THE_REQUEST "links -dump-charset "
    SecFilterSelective THE_REQUEST "links -dump-width "
    SecFilterSelective THE_REQUEST "links http:// "
    SecFilterSelective THE_REQUEST "links ftp:// "
    SecFilterSelective THE_REQUEST "links -source "
    SecFilterSelective THE_REQUEST "mkdir "
    SecFilterSelective THE_REQUEST "cd /tmp "
    SecFilterSelective THE_REQUEST "cd /var/tmp "
    SecFilterSelective THE_REQUEST "cd /etc/httpd/proxy "
    SecFilterSelective THE_REQUEST "/config.php?v=1&DIR "
    SecFilterSelective THE_REQUEST "/../../ "
    SecFilterSelective THE_REQUEST "&highlight=%2527%252E "
    SecFilterSelective THE_REQUEST "changedir=%2Ftmp%2F.php "
    
    # Very crude filters to prevent SQL injection attacks
    SecFilter "delete[[:space:]]+from"
    SecFilter "insert[[:space:]]+into"
    SecFilter "select.+from"
    
    # Weaker XSS protection but allows common HTML tags
    SecFilter "<[[:space:]]*script"
    
    # Prevent XSS atacks (HTML/Javascript injection)
    SecFilter "<(.|n)+>"
    </IfModule>

    Is there anything more i can do

    All experience people do reply

  7. #7
    Join Date
    Mar 2009
    Location
    /home/khunj
    Posts
    432
    The 2 files from the archive were 'stealth' and 'syslogd'.

    You can double-check all listening applications :

    Code:
    netstat -ntpa | grep LISTEN
    NinTechNet
    ★ NinjaFirewall : Web Application Firewall for PHP and WordPress.
    ★ NinjaMonitoring : Monitor your website for suspicious activities.

  8. #8
    Join Date
    Jul 2009
    Posts
    40
    Quote Originally Posted by khunj View Post
    The 2 files from the archive were 'stealth' and 'syslogd'.

    You can double-check all listening applications :

    Code:
    netstat -ntpa | grep LISTEN
    Here is the output -

    Code:
    # netstat -ntpa | grep LISTEN
    tcp        0      0 MY IP:3306         0.0.0.0:*                   LISTEN      4966/mysqld
    tcp        0      0 0.0.0.0:587                 0.0.0.0:*                   LISTEN      3193/exim
    tcp        0      0 0.0.0.0:14                  0.0.0.0:*                   LISTEN      4994/proftpd: (acce
    tcp        0      0 0.0.0.0:110                 0.0.0.0:*                   LISTEN      3241/vm-pop3d
    tcp        0      0 0.0.0.0:2222                0.0.0.0:*                   LISTEN      3179/directadmin
    tcp        0      0 127.0.0.1:783               0.0.0.0:*                   LISTEN      3198/spamd -d -c -m
    tcp        0      0 0.0.0.0:143                 0.0.0.0:*                   LISTEN      3099/xinetd
    tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      2875/portmap
    tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      21643/httpd
    tcp        0      0 MY IP :53           0.0.0.0:*                   LISTEN      3956/named
    tcp        0      0 MY IP:53           0.0.0.0:*                   LISTEN      3956/named
    tcp        0      0 127.0.0.1:53                0.0.0.0:*                   LISTEN      3956/named
    tcp        0      0 0.0.0.0:631                 0.0.0.0:*                   LISTEN      3012/cupsd
    tcp        0      0 127.0.0.1:953               0.0.0.0:*                   LISTEN      3956/named
    tcp        0      0 0.0.0.0:25                  0.0.0.0:*                   LISTEN      3193/exim
    tcp        0      0 0.0.0.0:443                 0.0.0.0:*                   LISTEN      21643/httpd
    tcp        0      0 0.0.0.0:956                 0.0.0.0:*                   LISTEN      2894/rpc.statd
    tcp        0      0 :::2200                     :::*                        LISTEN      3063/sshd
    tcp        0      0 ::1:953                     :::*                        LISTEN      3956/named

  9. #9
    Join Date
    Jul 2009
    Posts
    40
    Amazing, even after installing mod_security, my server is broken by excessive use of B/W

    Any suggestions please

  10. #10
    Join Date
    Jun 2006
    Location
    NYC
    Posts
    1,446
    Quote Originally Posted by Leaptopz View Post
    Hi All,

    I am undergoing "amazing" Bandwidth thief issue. My Server is using very high bandwidth from last 7 days -

    On Scanning server, i found following

    1) Roundcube mail script was compromised and used by some hacking group, so i deleted that directory all together
    2) I disabled Hotlinking all together
    3) I found "Live Chat" script also being compromised, and found movies (.dat files) in it amounting to 20 GB++ , i also deleted that directory
    4) I have checked for each and every file, with size more than 30 MB on server.
    5) I have also switched off FTP service (and there is no other user on server too)
    6) After all this, i even installed "mod_bandwidth" and limited bandwidth service to 10 KBPS across domain directory

    Then, i installed MRTG graphs to analyze the issue, and to my SURPRISE, nothing changed, it still used very high bandwidth, after few - few hours.

    http://img208.imageshack.us/img208/511/indexday.png

    ^^ the above image shows B/W goes upto 10 MBPS and thus breaks network.

    I am now tired and exhausted of this. Can someone here, please give an expert advice on this issue ?

    I hope some experience person will give an insight into this issue.
    Mod_security or not you've been highly compromised. Rebuild the box.

    Don't try to fix this, get the data off you need and do a fresh install.
    FiberPeer.Com | | REAL DDoS Protection | Cloud Hosting | VPS | Dedicated Servers | High Bandwidth Hosting | 1Gbps-10Gbps Unmetered
    FiberPeer DDoS Mitigation | ethProxy Upgraded! | 14-Years Experience | Emergency 24/7 Support
    Visit us @ www.fiberpeer.com

  11. #11
    Join Date
    Jul 2009
    Posts
    40
    Quote Originally Posted by serverorigin View Post
    Mod_security or not you've been highly compromised. Rebuild the box.

    Don't try to fix this, get the data off you need and do a fresh install.

    Rebuild will take enormous time, i would prefer to stick and find root cause of problem.

    Some experienced person, would help me

  12. #12
    Join Date
    May 2006
    Posts
    64
    Quote Originally Posted by Leaptopz View Post
    Rebuild will take enormous time, i would prefer to stick and find root cause of problem.

    Some experienced person, would help me
    impossible. checking each single (yes each) file on the server and comparing to a valid original will certainly take longer than a reinstallation.

    dont forget to check all the files in your backups for backdoors etc. and while you are at it look in the sql db if some new "admin" users were added


    on which port do you see the excess bandwidth usage? look with netstat which app that is, look in that app's log files whats being transferred, stop it, wait for the hackers to come back and exploit your system in another way, then realize you need to reinstall.

  13. #13
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,683
    Try installing CSF, which should lock down some of the strange port access and may help stop future account-level compromises. But I suspect that, since you're not sure what you're doing, you'd be better off hiring someone who knows what they're doing. The most likely method for fixing would be a reinstall which should be possible in 3-4 hours. Many datacentres will attach a second disk to allow you to rebuild, for instance. But get someone who knows what they're doing to help you, it's not trivial to fix root compromises.

  14. #14
    Join Date
    Apr 2005
    Posts
    535
    wondering why anyone hasn seen this in the netstat posted above :

    tcp 0 0 0.0.0.0:14 0.0.0.0:* LISTEN 4994/proftpd: (acce
    im sure the OP mentioned disabling ftp so why is their an ftp server active

  15. #15
    Join Date
    Jul 2009
    Posts
    40
    Quote Originally Posted by NWSTech View Post
    wondering why anyone hasn seen this in the netstat posted above :



    im sure the OP mentioned disabling ftp so why is their an ftp server active
    The server is again down
    Can you let me know, which command can i use to check, that what port is using max. b/w

  16. #16
    I have to agree with some of the comments in this thread. Don't try and deal with this problem yourself, you're just delaying the inevitable.

    Try to salvage any data you can and rebuild the box. Once you have the box rebuilt, please consult a professional to secure the server in order to reduce the chances of compromise in the future.

  17. #17
    Join Date
    Mar 2008
    Posts
    1,717
    Leaptopz: Yeah, that box has been rooted - ftp daemon on port 14, that's probably where all your bandwidth is going, your server is now a warez dump. There's a slim chance you could fix it if the guys that did it were noobs, but do you really want to take that chance?

    You're probably dealing with outages now, with that kind of usage. Deal with a 48 hour or so outage, backup and reinstall, and work from a clean state.

    Cleaning up after hackers basically means replacing anything that's suspect - a compromised user account means this can be done from the root account, but if the root account's been compromised everything on the machine is suspect.
    I used to run the oldest commercial Mumble host.

  18. #18
    Join Date
    Jul 2009
    Posts
    40
    Quote Originally Posted by fwaggle View Post
    Leaptopz: Yeah, that box has been rooted - ftp daemon on port 14, that's probably where all your bandwidth is going, your server is now a warez dump. There's a slim chance you could fix it if the guys that did it were noobs, but do you really want to take that chance?

    You're probably dealing with outages now, with that kind of usage. Deal with a 48 hour or so outage, backup and reinstall, and work from a clean state.

    Cleaning up after hackers basically means replacing anything that's suspect - a compromised user account means this can be done from the root account, but if the root account's been compromised everything on the machine is suspect.

    Hi, i myself have changed FTP port to 14.
    Even if i reinstall machine, i would be copying old files and DB, so i do not see that would make difference, so looking for root cause

  19. #19
    Join Date
    Mar 2009
    Location
    deep blue yonder
    Posts
    176
    Quote Originally Posted by Leaptopz View Post
    Can you let me know, which command can i use to check, that what port is using max. b/w
    Try using trafshow to map the high usage network streams to port numbers, then map the port numbers to the daemons in your netstat output.
    424 bits were harmed in the making of this signature.

  20. #20
    Join Date
    Mar 2009
    Location
    /home/khunj
    Posts
    432
    Your netstat output doesn't look bad, at least the IRC script isn't listening.

    Can you try this :

    Code:
    netstat -s
    The output will at least tell us which protocol eats up all your bandwidth.
    NinTechNet
    ★ NinjaFirewall : Web Application Firewall for PHP and WordPress.
    ★ NinjaMonitoring : Monitor your website for suspicious activities.

  21. #21
    Join Date
    Jul 2009
    Posts
    40
    Quote Originally Posted by khunj View Post
    Your netstat output doesn't look bad, at least the IRC script isn't listening.

    Can you try this :

    Code:
    netstat -s
    The output will at least tell us which protocol eats up all your bandwidth.

    Code:
    [root@ ]# netstat -s
    Ip:
        210791 total packets received
        10468 with invalid addresses
        0 forwarded
        0 incoming packets discarded
        197020 incoming packets delivered
        213926 requests sent out
        200 outgoing packets dropped
    Icmp:
        2146 ICMP messages received
        0 input ICMP message failed.
        ICMP input histogram:
            destination unreachable: 32
            echo requests: 2111
            echo replies: 3
        16216 ICMP messages sent
        0 ICMP messages failed
        ICMP output histogram:
            destination unreachable: 14105
            echo replies: 2111
    Tcp:
        102 active connections openings
        22516 passive connection openings
        4 failed connection attempts
        838 connection resets received
        15 connections established
        176490 segments received
        199644 segments send out
        4587 segments retransmited
        0 bad segments received.
        1409 resets sent
    Udp:
        3360 packets received
        14105 packets to unknown port received.
        0 packet receive errors
        3395 packets sent
    TcpExt:
        74 resets received for embryonic SYN_RECV sockets
        ArpFilter: 0
        18948 TCP sockets finished time wait in fast timer
        1267 delayed acks sent
        5 delayed acks further delayed because of locked socket
        Quick ack mode was activated 740 times
        33390 packets directly queued to recvmsg prequeue.
        2041 packets directly received from backlog
        26943924 packets directly received from prequeue
        4401 packets header predicted
        29222 packets header predicted and directly queued to user
        TCPPureAcks: 68949
        TCPHPAcks: 11928
        TCPRenoRecovery: 14
        TCPSackRecovery: 163
        TCPSACKReneging: 0
        TCPFACKReorder: 0
        TCPSACKReorder: 0
        TCPRenoReorder: 7
        TCPTSReorder: 0
        TCPFullUndo: 0
        TCPPartialUndo: 0
        TCPDSACKUndo: 16
        TCPLossUndo: 138
        TCPLoss: 121
        TCPLostRetransmit: 0
        TCPRenoFailures: 20
        TCPSackFailures: 285
        TCPLossFailures: 41
        TCPFastRetrans: 223
        TCPForwardRetrans: 4
        TCPSlowStartRetrans: 332
        TCPTimeouts: 1616
        TCPRenoRecoveryFail: 11
        TCPSackRecoveryFail: 36
        TCPSchedulerFailed: 0
        TCPRcvCollapsed: 0
        TCPDSACKOldSent: 829
        TCPDSACKOfoSent: 2
        TCPDSACKRecv: 198
        TCPDSACKOfoRecv: 0
        TCPAbortOnSyn: 0
        TCPAbortOnData: 49
        TCPAbortOnClose: 1
        TCPAbortOnMemory: 0
        TCPAbortOnTimeout: 299
        TCPAbortOnLinger: 0
        TCPAbortFailed: 0
        TCPMemoryPressures: 0

  22. #22
    Join Date
    Mar 2009
    Location
    /home/khunj
    Posts
    432
    Hard to tell, because it seems you have rebooted your server lately.
    Next time you are having problem, just run again the command, but before rebooting otherwise all fields will be cleared.

    Quote Originally Posted by Leaptopz View Post
    Code:
    [root@ ]# netstat -s
    Icmp:
        2146 ICMP messages received
        0 input ICMP message failed.
        ICMP input histogram:
            destination unreachable: 32
            echo requests: 2111
            echo replies: 3
        16216 ICMP messages sent
        0 ICMP messages failed
        ICMP output histogram:
            destination unreachable: 14105
            echo replies: 2111
    Udp:
        3360 packets received
        14105 packets to unknown port received.
        0 packet receive errors
        3395 packets sent
    You are receiving quite a lot of bogus UDP paquets.

    If you want to monitor traffic through one port, you can use iptables and create a rule which does nothing but counting bytes/packets.
    For instance, if you want to monitor incoming traffic on port 80 :

    Code:
    iptables -I INPUT -p tcp --dport 80
    Then run:
    Code:
    iptables -L -nv
    And you will get something like this, the first 2 columns showing the total of packets and bytes :

    Code:
    Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts     bytes    target     prot opt in     out     source                  destination         
    9739     6661K            tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:80
    NinTechNet
    ★ NinjaFirewall : Web Application Firewall for PHP and WordPress.
    ★ NinjaMonitoring : Monitor your website for suspicious activities.

Similar Threads

  1. magic quotes issue
    By sachin_linux in forum Hosting Security and Technology
    Replies: 1
    Last Post: 04-04-2008, 10:38 AM
  2. help me track down bandwidth thief - take 2
    By papi in forum Hosting Security and Technology
    Replies: 19
    Last Post: 03-22-2006, 09:37 AM
  3. Help me track down bandwidth thief
    By papi in forum Hosting Security and Technology
    Replies: 11
    Last Post: 02-16-2006, 09:30 AM
  4. AWSTATS Bandwidth totals different to Report Magic
    By sunnyD in forum Hosting Software and Control Panels
    Replies: 0
    Last Post: 02-24-2004, 08:20 AM
  5. Replies: 0
    Last Post: 08-08-2003, 09:32 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •