Results 1 to 19 of 19
  1. #1

    * Is this expected behavior when your server's pipe is overloaded?

    I have an unmetered 100mbps port with 10tb.com. Their servers are excellent, but when traffic exceeds 100mbps, the server becomes inaccessible, I can ping it but can't SSH to it or even reboot it from the main control panel.

    When this happens at later times, I have redirected traffic to another server, but the inaccessible server remains inaccessible. I have to ask for support to reboot it.

    This can't be normal, can it?

    Attached is image showing the spikes and outages.
    Attached Thumbnails Attached Thumbnails traffic.png  

  2. #2
    What is the server running, OS, applications?

    I wouldn't be surprised if it crashed from load everytime it got hit like that. Is there anything in dmesg after you reboot it?

  3. #3
    Join Date
    Feb 2004
    Location
    Atlanta, GA
    Posts
    5,627
    it would appear that something on your server is overloading it, possibly an exhaustion of available memory and swap causing the machine to come to a hault? Is it HTTP traffic? Is apache/iis overloading the server?

    This is not a scenario where network saturation would be causing issues. This is more of a load / overload / crashing on the server issue.

    Do you not have management on the server for them to check?

  4. #4
    Join Date
    Aug 2009
    Posts
    529
    Thats interesting can you let me know whats generating the traffic? Website ect. if it's a website do you mind proving the URL?

  5. #5
    Join Date
    Feb 2002
    Location
    New York, NY
    Posts
    4,612
    Maxing out the network port would cause packet loss and an increase in latency for the packets that do make it through, but it shouldn't cause the server to lock up. If it does, that's a problem.
    Scott Burns, President
    BQ Internet Corporation
    Remote Rsync and FTP backup solutions
    *** http://www.bqbackup.com/ ***

  6. #6
    Quote Originally Posted by bqinternet View Post
    Maxing out the network port would cause packet loss and an increase in latency for the packets that do make it through, but it shouldn't cause the server to lock up. If it does, that's a problem.
    Was thinking the same

  7. #7
    Join Date
    Oct 2007
    Location
    Reading, UK
    Posts
    601
    This is going to relate to what ever is causing the bandwidth spike, but the spike is not the crash, the cause of the spike is the crash. Please provide details of your server set up and what you are hosting so we can all help you better
    Toby H
    Infrastructure Consultant
    SolutionFire.co.uk
    cPanel | VMware | XenServer | cPanel | 3PAR | R1soft

  8. #8
    It's hosting a number of websites, but only since the addition of a online karaoke site did this happen. The karaoke site is in Vietnamese at diepkhuc [dot] com

    Most of the traffic of the site is Apache downloading MP3 karaoke tracks and Red5 streaming.

    Server uptime: 19 hours 48 minutes 58 seconds
    Total accesses: 2771962 - Total Traffic: 378.6 GB
    CPU Usage: u375.54 s24.67 cu0 cs0 - .561% CPU load
    38.9 requests/sec - 5.4 MB/second - 143.2 kB/request
    181 requests currently being processed, 35 idle workers
    The server is a Quad Xeon 5520 with 6GB RAM running CentOS. At peak time, cumulative CPU usage is no more than 25%. And at peak time there are about 500 Red5 streaming clients. Unfortunately we don't have any management plan

    The thing is the server is fine if bandwidth comes close but doesn't hit 100Mbps, such as this morning (attached usage capture below). I will check dmesg next time, thanks.
    Attached Thumbnails Attached Thumbnails traffic.png  
    Last edited by ken107; 11-09-2009 at 10:33 AM. Reason: add attachment

  9. #9
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,571
    You're probably getting load spikes which consume all your available RAM. the kernel then starts killing off processes which is why you can no longer access the box but able to ping it (kernel still alive). If you do some optimization you could probably keep it from crashing, but either way looks like you're throwing more traffic than it can safely handle.
    Fast Serv Networks, LLC | AS29889 | Fully Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  10. #10
    Join Date
    Oct 2007
    Location
    Reading, UK
    Posts
    601
    Hi,

    Have you got any kind of monitoring set up on this server?

    I would advise you to set up health monitoring from www.wormly.com on there so you can get some information back as to exactly what is going on when the system crashes. Also, if set correctly you could get it to sms alert you of abnormal traffic / load / ram use etc.
    Toby H
    Infrastructure Consultant
    SolutionFire.co.uk
    cPanel | VMware | XenServer | cPanel | 3PAR | R1soft

  11. #11
    Join Date
    Mar 2008
    Posts
    1,717
    Quote Originally Posted by FastServ View Post
    You're probably getting load spikes which consume all your available RAM. the kernel then starts killing off processes which is why you can no longer access the box but able to ping it (kernel still alive).
    Or he has too much swap. The kernel only starts killing things if it runs out of swap - if you have too much, it just spends all it's time paging things in and out of swap, which makes the whole thing very unresponsive.

    You are correct though - I believe the kernel won't need to wait for virtual memory to answer pings (probably get the same behavior for RSTs as well, if you don't have a firewall dropping them connecting to a non-listening port probably answers with an RST instantly). IMHO it's most likely a RAM exhaustion issue.
    I used to run the oldest commercial Mumble host.

  12. #12
    Join Date
    Mar 2009
    Posts
    2,218
    Quote Originally Posted by ken107 View Post
    but only since the addition of a online karaoke site did this happen. The karaoke site is in Vietnamese at diepkhuc [dot] com

    Most of the traffic of the site is Apache downloading MP3 karaoke tracks and Red5 streaming.

    The server is a Quad Xeon 5520 with 6GB RAM running CentOS. At peak time, cumulative CPU usage is no more than 25%. And at peak time there are about 500 Red5 streaming clients. Unfortunately we don't have any management plan

    The thing is the server is fine if bandwidth comes close but doesn't hit 100Mbps, such as this morning (attached usage capture below). I will check dmesg next time, thanks.
    I'd say the streaming connections are simply swamping the network connection.

    The Karaoke site has videos; the video I checked had an average traffic rate of over 600,000 bits per second.

    Do you know the data rate for the other streams? Then you can work out the average load for the 500 streaming users.

  13. #13
    Join Date
    Oct 2007
    Location
    Reading, UK
    Posts
    601
    Quote Originally Posted by tim2718281 View Post
    I'd say the streaming connections are simply swamping the network connection.

    The Karaoke site has videos; the video I checked had an average traffic rate of over 600,000 bits per second.

    Do you know the data rate for the other streams? Then you can work out the average load for the 500 streaming users.
    Even so, it shouldn't bring down the server, just may it suffer packet loss.

    The best way to diagnose this is to monitor the network traffic, memory use, swap use and load averages for a while and see what is going on.

    Toby H
    Infrastructure Consultant
    SolutionFire.co.uk
    cPanel | VMware | XenServer | cPanel | 3PAR | R1soft

  14. #14
    Join Date
    Mar 2009
    Posts
    2,218
    Quote Originally Posted by Toby H View Post
    Even so, it shouldn't bring down the server
    Maybe it doesn't, but just makes connecting via the network difficult or practically impossible.

    I guess the OP could open a terminal connection *before* the problem occurs, and keep it open.

  15. #15
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,571
    Quote Originally Posted by fwaggle View Post
    Or he has too much swap. The kernel only starts killing things if it runs out of swap - if you have too much, it just spends all it's time paging things in and out of swap, which makes the whole thing very unresponsive.

    You are correct though - I believe the kernel won't need to wait for virtual memory to answer pings (probably get the same behavior for RSTs as well, if you don't have a firewall dropping them connecting to a non-listening port probably answers with an RST instantly). IMHO it's most likely a RAM exhaustion issue.
    I agree 100%. Try reducing swap and increasing physical memory, perhaps double at 12Gb.

    I didn't realize you were streaming. When you max your port out, all those connections start buffering which uses more memory on the server for every connection and leads to swap which leads to boom crash bang .
    Fast Serv Networks, LLC | AS29889 | Fully Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  16. #16
    Join Date
    Nov 2007
    Posts
    62
    I have pushed around 60MB/s (600mbit effective) on my 10TB server and no problems. Same configuration as you. I use nginx instead of Apache though. Load gets fairly high (2.50-3) probably due to disk I/O but no crashes.

  17. #17
    Join Date
    Aug 2002
    Location
    Seattle
    Posts
    5,512
    I wouldn't rule out severe packet loss, typically when we see a port reach critical mass (eg. 98+ Mbps on a FE) it gets pretty nasty and connections do begin to drop and timeout.

  18. #18
    Join Date
    Mar 2009
    Posts
    2,218
    Quote Originally Posted by IRCCo Jeff View Post
    I wouldn't rule out severe packet loss, typically when we see a port reach critical mass (eg. 98+ Mbps on a FE) it gets pretty nasty and connections do begin to drop and timeout.
    I guess you can configure Linux to prevent the streaming apps consuming all the bandwidth.

    I don't know enough about this; I know you use the tc command to set QoS parameters.

    It would probably be very useful if someone who knows that they are doing could explain it! (For example, how to limit streaming to 90mbps.)

  19. #19
    Join Date
    Nov 2006
    Location
    Karachi, Pakistan
    Posts
    1,349
    You need to fine tune the apache max clients setting. You've set it too high causing apache to use up all your RAM and then starts digging into swap space. At that point load goes up and your server locks up. A good rule of thumb for max clients is 100 per GB or RAM. For 6GB I would set it to 600 and then fine tune it very carefully.

Similar Threads

  1. IE 7 strange behavior
    By sergeyka in forum Hosting Security and Technology
    Replies: 1
    Last Post: 08-28-2009, 11:53 AM
  2. Overloaded?
    By Deca in forum Web Hosting
    Replies: 15
    Last Post: 03-03-2009, 02:27 PM
  3. Have I overloaded my vps?
    By Buffy807 in forum VPS Hosting
    Replies: 4
    Last Post: 02-21-2008, 05:21 PM
  4. Am I overloaded?
    By lalaland in forum Dedicated Server
    Replies: 7
    Last Post: 12-28-2007, 02:22 AM
  5. Vps Overloaded
    By adrianjare in forum VPS Hosting
    Replies: 9
    Last Post: 11-04-2006, 12:30 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •