Results 1 to 6 of 6
  1. #1

    CPanel - server hangs - swpused 100%

    Hello all,

    For some reason once in a while when I wake up in the morning I find my server frozen.

    'sar -r' showed:

    Code:
    08:20:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
    08:30:01 AM    222456   3836460     94.52    132004   2817320   8313500        96      0.00         0
    08:40:01 AM    158160   3900756     96.10    137372   2833528   8313500        96      0.00         0
    08:50:01 AM    192340   3866576     95.26    142124   2844744   8313500        96      0.00         0
    09:00:01 AM     13152   4045764     99.68     18560    420240   8309212      4384      0.05      4288
    09:10:02 AM     12680   4046236     99.69     16232    400024   1034632   7278964     87.55     97472
    09:41:09 AM   3362284    696632     17.16      1788     18412         0   8313596    100.00     56852
    09:51:22 AM     12788   4046128     99.68      1424     54428         0   8313596    100.00     19488
    Average:       197831   3861085     95.13     85435   2744956   7890186    423410      5.09      3533
    
    10:03:54 AM       LINUX RESTART
    
    10:10:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
    10:20:01 AM   2583340   1475576     36.35     66780    823564   8313596         0      0.00         0
    Average:      2583340   1475576     36.35     66780    823564   8313596         0      0.00         0
    /var/log/messages says:

    Code:
    Apr 13 08:54:53 server01 named[4542]: client 217.69.181.175#2: view external: error sending response: host unreachable
    Apr 13 08:54:59 server01 last message repeated 3 times
    Apr 13 08:55:00 server01 named[4542]: client 78.89.191.23#1: view external: error sending response: host unreachable
    Apr 13 08:55:01 server01 named[4542]: client 78.89.191.23#1: view external: error sending response: host unreachable
    Apr 13 08:55:05 server01 named[4542]: client 217.69.181.175#1: view external: error sending response: host unreachable
    Apr 13 08:55:18 server01 named[4542]: client 217.69.181.175#1: view external: error sending response: host unreachable
    Apr 13 08:57:24 server01 named[4542]: client 217.69.181.175#2: view external: error sending response: host unreachable
    Apr 13 08:57:43 server01 last message repeated 4 times
    Apr 13 09:37:05 server01 kernel: couriertls invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
    Apr 13 09:37:08 server01 kernel: 
    Apr 13 09:37:08 server01 kernel: Call Trace:
    Apr 13 09:37:09 server01 kernel:  [<ffffffff800c3a6e>] out_of_memory+0x8e/0x2f5
    Apr 13 09:37:09 server01 kernel:  [<ffffffff8000f2eb>] __alloc_pages+0x245/0x2ce
    Apr 13 09:37:09 server01 kernel:  [<ffffffff80012a62>] __do_page_cache_readahead+0x95/0x1d9
    Apr 13 09:37:09 server01 kernel:  [<ffffffff800c87cd>] zone_statistics+0x3e/0x6d
    Apr 13 09:37:09 server01 kernel:  [<ffffffff8000a87f>] get_page_from_freelist+0x377/0x3fa
    Apr 13 09:37:09 server01 kernel:  [<ffffffff800c87cd>] zone_statistics+0x3e/0x6d
    Apr 13 09:37:09 server01 kernel:  [<ffffffff800****1>] filemap_nopage+0x148/0x322
    Apr 13 09:37:09 server01 kernel:  [<ffffffff80008863>] __handle_mm_fault+0x1f8/0xe5c
    Apr 13 09:37:09 server01 kernel:  [<ffffffff80066b9a>] do_page_fault+0x4cb/0x830
    Apr 13 09:38:42 server01 kernel:  [<ffffffff8001c87b>] vma_link+0x70/0xfd
    Apr 13 09:39:38 server01 named[4542]: client 74.125.44.78#36164: view external: error sending response: host unreachable
    Apr 13 09:40:01 server01 kernel:  [<ffffffff8000de64>] do_mmap_pgoff+0x66c/0x7d7
    Apr 13 09:40:46 server01 kernel:  [<ffffffff8005dde9>] error_exit+0x0/0x84
    Apr 13 09:41:03 server01 kernel: 
    Apr 13 09:41:03 server01 kernel: Mem-info:
    Apr 13 09:40:38 server01 named[4542]: client 210.113.99.6#46477: view external: error sending response: host unreachable
    Apr 13 09:41:05 server01 kernel: Node 0 DMA per-cpu:
    Apr 13 09:41:12 server01 kernel: cpu 0 hot: high 0, batch 1 used:0
    Apr 13 09:41:13 server01 kernel: cpu 0 cold: high 0, batch 1 used:0
    Apr 13 09:41:14 server01 kernel: cpu 1 hot: high 0, batch 1 used:0
    Apr 13 09:41:14 server01 kernel: cpu 1 cold: high 0, batch 1 used:0
    Apr 13 09:41:15 server01 kernel: Node 0 DMA32 per-cpu:
    Apr 13 09:41:15 server01 kernel: cpu 0 hot: high 186, batch 31 used:1
    Apr 13 09:41:15 server01 kernel: cpu 0 cold: high 62, batch 15 used:30
    Apr 13 09:41:15 server01 kernel: cpu 1 hot: high 186, batch 31 used:0
    Apr 13 09:41:15 server01 kernel: cpu 1 cold: high 62, batch 15 used:0
    Apr 13 09:41:16 server01 kernel: Node 0 Normal per-cpu: empty
    Apr 13 09:41:16 server01 kernel: Node 0 HighMem per-cpu: empty
    Apr 13 09:41:16 server01 kernel: Node 1 DMA per-cpu: empty
    Apr 13 09:41:18 server01 named[4542]: client 74.125.44.78#17405: view external: error sending response: host unreachable
    Apr 13 09:41:18 server01 kernel: Node 1 DMA32 per-cpu:
    Apr 13 09:41:18 server01 kernel: cpu 0 hot: high 186, batch 31 used:0
    Apr 13 09:41:18 server01 kernel: cpu 0 cold: high 62, batch 15 used:0
    Apr 13 09:41:18 server01 kernel: cpu 1 hot: high 186, batch 31 used:17
    Apr 13 09:41:19 server01 kernel: cpu 1 cold: high 62, batch 15 used:44
    Apr 13 09:41:19 server01 kernel: Node 1 Normal per-cpu: empty
    Apr 13 09:41:19 server01 kernel: Node 1 HighMem per-cpu: empty
    Apr 13 09:41:19 server01 kernel: Free pages:       10112kB (0kB HighMem)
    Apr 13 09:41:19 server01 kernel: Active:507540 inactive:456284 dirty:0 writeback:0 unstable:0 free:2528 slab:8223 mapped-file:1 mapp
    ed-anon:965692 pagetables:26102
    Apr 13 09:41:19 server01 kernel: Node 0 DMA free:2100kB min:20kB low:24kB high:28kB active:0kB inactive:0kB present:10692kB pages_sc
    anned:0 all_unreclaimable? yes
    Apr 13 09:41:19 server01 kernel: lowmem_reserve[]: 0 2004 2004 2004
    Apr 13 09:41:19 server01 kernel: Node 0 DMA32 free:4064kB min:4068kB low:5084kB high:6100kB active:976184kB inactive:972368kB presen
    t:2052320kB pages_scanned:6413010 all_unreclaimable? yes
    Apr 13 09:41:19 server01 kernel: lowmem_reserve[]: 0 0 0 0
    Apr 13 09:41:19 server01 kernel: Node 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0
     all_unreclaimable? no
    Apr 13 09:41:19 server01 kernel: lowmem_reserve[]: 0 0 0 0
    Apr 13 09:41:20 server01 kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_sc
    anned:0 all_unreclaimable? no
    Apr 13 09:41:20 server01 kernel: lowmem_reserve[]: 0 0 0 0
    Apr 13 09:41:20 server01 kernel: Node 1 DMA free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 al
    l_unreclaimable? no
    Apr 13 09:41:20 server01 kernel: lowmem_reserve[]: 0 1956 1956 1956
    Apr 13 09:41:20 server01 kernel: Node 1 DMA32 free:3948kB min:3972kB low:4964kB high:5956kB active:1053976kB inactive:852768kB prese
    nt:2003272kB pages_scanned:3940088 all_unreclaimable? yes
    Apr 13 09:41:20 server01 kernel: lowmem_reserve[]: 0 0 0 0
    Apr 13 09:41:21 server01 named[4542]: client 210.113.99.6#47920: view external: error sending response: host unreachable
    Apr 13 09:41:21 server01 kernel: Node 1 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0
     all_unreclaimable? no
    Apr 13 09:41:22 server01 kernel: lowmem_reserve[]: 0 0 0 0
    Apr 13 09:41:24 server01 kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_sc
    anned:0 all_unreclaimable? no
    Apr 13 09:41:24 server01 named[4542]: client 67.195.115.249#59598: view external: error sending response: host unreachable
    Apr 13 09:41:25 server01 kernel: lowmem_reserve[]: 0 0 0 0
    Apr 13 09:41:25 server01 kernel: Node 0 DMA: 3*4kB 3*8kB 1*16kB 4*32kB 4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2
    100kB
    Apr 13 09:41:26 server01 named[4542]: client 210.113.99.6#61090: view external: error sending response: host unreachable
    Apr 13 09:41:26 server01 kernel: Node 0 DMA32: 12*4kB 16*8kB 3*16kB 2*32kB 3*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB
     = 4064kB
    Apr 13 09:41:26 server01 named[4542]: client 210.113.99.6#5115: view external: error sending response: host unreachable
    Apr 13 09:41:26 server01 kernel: Node 0 Normal: empty
    Apr 13 09:41:26 server01 named[4542]: client 210.113.99.6#19259: view external: error sending response: host unreachable
    Apr 13 09:41:26 server01 kernel: Node 0 HighMem: empty
    Apr 13 09:41:27 server01 kernel: Node 1 DMA: empty
    Apr 13 09:41:27 server01 kernel: Node 1 DMA32: 7*4kB 6*8kB 2*16kB 0*32kB 4*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB =
     3948kB
    Apr 13 09:41:27 server01 kernel: Node 1 Normal: empty
    Apr 13 09:41:28 server01 kernel: Node 1 HighMem: empty
    Apr 13 09:41:28 server01 kernel: 2838 pagecache pages
    Apr 13 09:41:28 server01 kernel: Swap cache: add 2381945, delete 2381942, find 215710/240774, race 0+27
    Apr 13 09:41:28 server01 kernel: Free swap  = 0kB
    Apr 13 09:41:29 server01 kernel: Total swap = 8313596kB
    Apr 13 09:41:29 server01 kernel: Free swap:            0kB
    Apr 13 09:41:29 server01 kernel: 1032048 pages of RAM
    Apr 13 09:41:30 server01 kernel: 17319 reserved pages
    Apr 13 09:41:30 server01 kernel: 184544 pages shared
    Apr 13 09:41:30 server01 kernel: 3 pages swap cached
    Apr 13 09:41:30 server01 kernel: Out of memory: Killed process 17640 (httpd).
    Apr 13 09:41:38 server01 named[4542]: client 210.113.99.6#24967: view external: error sending response: host unreachable
    Apr 13 09:41:38 server01 named[4542]: client 74.125.44.78#24426: view external: error sending response: host unreachable
    Apr 13 09:41:38 server01 named[4542]: client 210.113.99.6#9658: view external: error sending response: host unreachable
    Apr 13 09:41:38 server01 named[4542]: client 210.113.99.6#30102: view external: error sending response: host unreachable
    Apr 13 09:41:38 server01 named[4542]: client 195.195.4.178#58981: view external: error sending response: host unreachable
    Apr 13 09:41:38 server01 named[4542]: client 195.195.4.178#34282: view external: error sending response: host unreachable
    Apr 13 09:41:38 server01 named[4542]: client 67.195.115.249#49669: view external: error sending response: host unreachable
    So what is causing the server to hang, and what do I need to do to prevent that happening?

    The server does have plenty of memory and CPU power to cope with the running applications and the load.

    Many thanks

  2. #2
    Join Date
    Feb 2004
    Location
    Toronto
    Posts
    2,308
    Memory / swap full usually means a malicious app is being run on your server.

    One of the more common cause is a proxy script. Run top and see what file is being called.

    Also check /tmp for any unusual files.
    VimHost >> 30 Days Backup | cPanel + LiteSpeed + JetBackup | DMCA FREE!
    20 Years in business ~ Premium Hosting in Toronto, Canada ~ 151 Front Street (Canadian owned and operated)

  3. #3
    I'd suggest running a ps -aux

    Use that to look for services/scripts that are using up a lot of memory. If they're malicious kill them off and secure the server. If they're regular services I would try shutting down the service, kill off any hanging processes, and then restart the service. This will usually free up memory.

  4. #4
    This only tends to happen overnight. All other times the server seems to behave normally.

    according to ps -aux there don't seem to be any malicious script running nor is there any particular process that is taking up too much CPU or RAM. It is running perfectly normal as always. Only once in a while something happens to it that brings it to a halt.

    Is there anyway I can find out what suddenly caused swap to be utilised by 100% between 9am to 9:40 am?

    /tmp seems to have all the normal files:

    aquota.user*
    eaccelerator/
    .htaccess
    .ICE-unix/
    lost+found/
    mt-throttle.db
    mysql.sock@
    lots of session files
    spamd-20109-init/
    ssh-cESyHl9190/

    Any more clues?

  5. #5
    I would setup a system snapshot that runs every minute or so and logs the current usage, processes, connections, etc... When this happens again you can view the logs to determine the exact cause.

  6. #6
    how is the mysql service holding up

    try mysqladmin proc

Similar Threads

  1. apache hangs cpu 100% utilization many http requests
    By eugene2006 in forum VPS Hosting
    Replies: 8
    Last Post: 11-19-2006, 08:34 PM
  2. cpanel hangs after entering username and password
    By juworld in forum Hosting Software and Control Panels
    Replies: 2
    Last Post: 09-27-2005, 10:49 AM
  3. Server Hangs on Reboot - CentOS + CPanel
    By VanHost in forum Hosting Security and Technology
    Replies: 5
    Last Post: 01-31-2005, 12:41 PM
  4. I try to upgrade cPanel, but it hangs..
    By NOCTroll in forum Dedicated Server
    Replies: 6
    Last Post: 12-25-2004, 05:50 AM
  5. Server hangs up
    By singhk in forum Hosting Security and Technology
    Replies: 6
    Last Post: 04-26-2004, 08:15 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •