Results 1 to 6 of 6
-
04-13-2010, 06:33 AM #1Newbie
- Join Date
- Apr 2010
- Posts
- 16
CPanel - server hangs - swpused 100%
Hello all,
For some reason once in a while when I wake up in the morning I find my server frozen.
'sar -r' showed:
Code:08:20:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad 08:30:01 AM 222456 3836460 94.52 132004 2817320 8313500 96 0.00 0 08:40:01 AM 158160 3900756 96.10 137372 2833528 8313500 96 0.00 0 08:50:01 AM 192340 3866576 95.26 142124 2844744 8313500 96 0.00 0 09:00:01 AM 13152 4045764 99.68 18560 420240 8309212 4384 0.05 4288 09:10:02 AM 12680 4046236 99.69 16232 400024 1034632 7278964 87.55 97472 09:41:09 AM 3362284 696632 17.16 1788 18412 0 8313596 100.00 56852 09:51:22 AM 12788 4046128 99.68 1424 54428 0 8313596 100.00 19488 Average: 197831 3861085 95.13 85435 2744956 7890186 423410 5.09 3533 10:03:54 AM LINUX RESTART 10:10:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad 10:20:01 AM 2583340 1475576 36.35 66780 823564 8313596 0 0.00 0 Average: 2583340 1475576 36.35 66780 823564 8313596 0 0.00 0
Code:Apr 13 08:54:53 server01 named[4542]: client 217.69.181.175#2: view external: error sending response: host unreachable Apr 13 08:54:59 server01 last message repeated 3 times Apr 13 08:55:00 server01 named[4542]: client 78.89.191.23#1: view external: error sending response: host unreachable Apr 13 08:55:01 server01 named[4542]: client 78.89.191.23#1: view external: error sending response: host unreachable Apr 13 08:55:05 server01 named[4542]: client 217.69.181.175#1: view external: error sending response: host unreachable Apr 13 08:55:18 server01 named[4542]: client 217.69.181.175#1: view external: error sending response: host unreachable Apr 13 08:57:24 server01 named[4542]: client 217.69.181.175#2: view external: error sending response: host unreachable Apr 13 08:57:43 server01 last message repeated 4 times Apr 13 09:37:05 server01 kernel: couriertls invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Apr 13 09:37:08 server01 kernel: Apr 13 09:37:08 server01 kernel: Call Trace: Apr 13 09:37:09 server01 kernel: [<ffffffff800c3a6e>] out_of_memory+0x8e/0x2f5 Apr 13 09:37:09 server01 kernel: [<ffffffff8000f2eb>] __alloc_pages+0x245/0x2ce Apr 13 09:37:09 server01 kernel: [<ffffffff80012a62>] __do_page_cache_readahead+0x95/0x1d9 Apr 13 09:37:09 server01 kernel: [<ffffffff800c87cd>] zone_statistics+0x3e/0x6d Apr 13 09:37:09 server01 kernel: [<ffffffff8000a87f>] get_page_from_freelist+0x377/0x3fa Apr 13 09:37:09 server01 kernel: [<ffffffff800c87cd>] zone_statistics+0x3e/0x6d Apr 13 09:37:09 server01 kernel: [<ffffffff800****1>] filemap_nopage+0x148/0x322 Apr 13 09:37:09 server01 kernel: [<ffffffff80008863>] __handle_mm_fault+0x1f8/0xe5c Apr 13 09:37:09 server01 kernel: [<ffffffff80066b9a>] do_page_fault+0x4cb/0x830 Apr 13 09:38:42 server01 kernel: [<ffffffff8001c87b>] vma_link+0x70/0xfd Apr 13 09:39:38 server01 named[4542]: client 74.125.44.78#36164: view external: error sending response: host unreachable Apr 13 09:40:01 server01 kernel: [<ffffffff8000de64>] do_mmap_pgoff+0x66c/0x7d7 Apr 13 09:40:46 server01 kernel: [<ffffffff8005dde9>] error_exit+0x0/0x84 Apr 13 09:41:03 server01 kernel: Apr 13 09:41:03 server01 kernel: Mem-info: Apr 13 09:40:38 server01 named[4542]: client 210.113.99.6#46477: view external: error sending response: host unreachable Apr 13 09:41:05 server01 kernel: Node 0 DMA per-cpu: Apr 13 09:41:12 server01 kernel: cpu 0 hot: high 0, batch 1 used:0 Apr 13 09:41:13 server01 kernel: cpu 0 cold: high 0, batch 1 used:0 Apr 13 09:41:14 server01 kernel: cpu 1 hot: high 0, batch 1 used:0 Apr 13 09:41:14 server01 kernel: cpu 1 cold: high 0, batch 1 used:0 Apr 13 09:41:15 server01 kernel: Node 0 DMA32 per-cpu: Apr 13 09:41:15 server01 kernel: cpu 0 hot: high 186, batch 31 used:1 Apr 13 09:41:15 server01 kernel: cpu 0 cold: high 62, batch 15 used:30 Apr 13 09:41:15 server01 kernel: cpu 1 hot: high 186, batch 31 used:0 Apr 13 09:41:15 server01 kernel: cpu 1 cold: high 62, batch 15 used:0 Apr 13 09:41:16 server01 kernel: Node 0 Normal per-cpu: empty Apr 13 09:41:16 server01 kernel: Node 0 HighMem per-cpu: empty Apr 13 09:41:16 server01 kernel: Node 1 DMA per-cpu: empty Apr 13 09:41:18 server01 named[4542]: client 74.125.44.78#17405: view external: error sending response: host unreachable Apr 13 09:41:18 server01 kernel: Node 1 DMA32 per-cpu: Apr 13 09:41:18 server01 kernel: cpu 0 hot: high 186, batch 31 used:0 Apr 13 09:41:18 server01 kernel: cpu 0 cold: high 62, batch 15 used:0 Apr 13 09:41:18 server01 kernel: cpu 1 hot: high 186, batch 31 used:17 Apr 13 09:41:19 server01 kernel: cpu 1 cold: high 62, batch 15 used:44 Apr 13 09:41:19 server01 kernel: Node 1 Normal per-cpu: empty Apr 13 09:41:19 server01 kernel: Node 1 HighMem per-cpu: empty Apr 13 09:41:19 server01 kernel: Free pages: 10112kB (0kB HighMem) Apr 13 09:41:19 server01 kernel: Active:507540 inactive:456284 dirty:0 writeback:0 unstable:0 free:2528 slab:8223 mapped-file:1 mapp ed-anon:965692 pagetables:26102 Apr 13 09:41:19 server01 kernel: Node 0 DMA free:2100kB min:20kB low:24kB high:28kB active:0kB inactive:0kB present:10692kB pages_sc anned:0 all_unreclaimable? yes Apr 13 09:41:19 server01 kernel: lowmem_reserve[]: 0 2004 2004 2004 Apr 13 09:41:19 server01 kernel: Node 0 DMA32 free:4064kB min:4068kB low:5084kB high:6100kB active:976184kB inactive:972368kB presen t:2052320kB pages_scanned:6413010 all_unreclaimable? yes Apr 13 09:41:19 server01 kernel: lowmem_reserve[]: 0 0 0 0 Apr 13 09:41:19 server01 kernel: Node 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 13 09:41:19 server01 kernel: lowmem_reserve[]: 0 0 0 0 Apr 13 09:41:20 server01 kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_sc anned:0 all_unreclaimable? no Apr 13 09:41:20 server01 kernel: lowmem_reserve[]: 0 0 0 0 Apr 13 09:41:20 server01 kernel: Node 1 DMA free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 al l_unreclaimable? no Apr 13 09:41:20 server01 kernel: lowmem_reserve[]: 0 1956 1956 1956 Apr 13 09:41:20 server01 kernel: Node 1 DMA32 free:3948kB min:3972kB low:4964kB high:5956kB active:1053976kB inactive:852768kB prese nt:2003272kB pages_scanned:3940088 all_unreclaimable? yes Apr 13 09:41:20 server01 kernel: lowmem_reserve[]: 0 0 0 0 Apr 13 09:41:21 server01 named[4542]: client 210.113.99.6#47920: view external: error sending response: host unreachable Apr 13 09:41:21 server01 kernel: Node 1 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 13 09:41:22 server01 kernel: lowmem_reserve[]: 0 0 0 0 Apr 13 09:41:24 server01 kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_sc anned:0 all_unreclaimable? no Apr 13 09:41:24 server01 named[4542]: client 67.195.115.249#59598: view external: error sending response: host unreachable Apr 13 09:41:25 server01 kernel: lowmem_reserve[]: 0 0 0 0 Apr 13 09:41:25 server01 kernel: Node 0 DMA: 3*4kB 3*8kB 1*16kB 4*32kB 4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2 100kB Apr 13 09:41:26 server01 named[4542]: client 210.113.99.6#61090: view external: error sending response: host unreachable Apr 13 09:41:26 server01 kernel: Node 0 DMA32: 12*4kB 16*8kB 3*16kB 2*32kB 3*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4064kB Apr 13 09:41:26 server01 named[4542]: client 210.113.99.6#5115: view external: error sending response: host unreachable Apr 13 09:41:26 server01 kernel: Node 0 Normal: empty Apr 13 09:41:26 server01 named[4542]: client 210.113.99.6#19259: view external: error sending response: host unreachable Apr 13 09:41:26 server01 kernel: Node 0 HighMem: empty Apr 13 09:41:27 server01 kernel: Node 1 DMA: empty Apr 13 09:41:27 server01 kernel: Node 1 DMA32: 7*4kB 6*8kB 2*16kB 0*32kB 4*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3948kB Apr 13 09:41:27 server01 kernel: Node 1 Normal: empty Apr 13 09:41:28 server01 kernel: Node 1 HighMem: empty Apr 13 09:41:28 server01 kernel: 2838 pagecache pages Apr 13 09:41:28 server01 kernel: Swap cache: add 2381945, delete 2381942, find 215710/240774, race 0+27 Apr 13 09:41:28 server01 kernel: Free swap = 0kB Apr 13 09:41:29 server01 kernel: Total swap = 8313596kB Apr 13 09:41:29 server01 kernel: Free swap: 0kB Apr 13 09:41:29 server01 kernel: 1032048 pages of RAM Apr 13 09:41:30 server01 kernel: 17319 reserved pages Apr 13 09:41:30 server01 kernel: 184544 pages shared Apr 13 09:41:30 server01 kernel: 3 pages swap cached Apr 13 09:41:30 server01 kernel: Out of memory: Killed process 17640 (httpd). Apr 13 09:41:38 server01 named[4542]: client 210.113.99.6#24967: view external: error sending response: host unreachable Apr 13 09:41:38 server01 named[4542]: client 74.125.44.78#24426: view external: error sending response: host unreachable Apr 13 09:41:38 server01 named[4542]: client 210.113.99.6#9658: view external: error sending response: host unreachable Apr 13 09:41:38 server01 named[4542]: client 210.113.99.6#30102: view external: error sending response: host unreachable Apr 13 09:41:38 server01 named[4542]: client 195.195.4.178#58981: view external: error sending response: host unreachable Apr 13 09:41:38 server01 named[4542]: client 195.195.4.178#34282: view external: error sending response: host unreachable Apr 13 09:41:38 server01 named[4542]: client 67.195.115.249#49669: view external: error sending response: host unreachable
The server does have plenty of memory and CPU power to cope with the running applications and the load.
Many thanks
-
04-13-2010, 06:40 AM #2Web Hosting Master
- Join Date
- Feb 2004
- Location
- Toronto
- Posts
- 2,308
Memory / swap full usually means a malicious app is being run on your server.
One of the more common cause is a proxy script. Run top and see what file is being called.
Also check /tmp for any unusual files.VimHost >> 30 Days Backup | cPanel + LiteSpeed + JetBackup | DMCA FREE!
20 Years in business ~ Premium Hosting in Toronto, Canada ~ 151 Front Street (Canadian owned and operated)
-
04-13-2010, 08:10 AM #3Junior Guru Wannabe
- Join Date
- Apr 2010
- Posts
- 65
I'd suggest running a ps -aux
Use that to look for services/scripts that are using up a lot of memory. If they're malicious kill them off and secure the server. If they're regular services I would try shutting down the service, kill off any hanging processes, and then restart the service. This will usually free up memory.
-
04-13-2010, 08:29 AM #4Newbie
- Join Date
- Apr 2010
- Posts
- 16
This only tends to happen overnight. All other times the server seems to behave normally.
according to ps -aux there don't seem to be any malicious script running nor is there any particular process that is taking up too much CPU or RAM. It is running perfectly normal as always. Only once in a while something happens to it that brings it to a halt.
Is there anyway I can find out what suddenly caused swap to be utilised by 100% between 9am to 9:40 am?
/tmp seems to have all the normal files:
aquota.user*
eaccelerator/
.htaccess
.ICE-unix/
lost+found/
mt-throttle.db
mysql.sock@
lots of session files
spamd-20109-init/
ssh-cESyHl9190/
Any more clues?
-
04-13-2010, 08:44 AM #5Junior Guru Wannabe
- Join Date
- Apr 2010
- Posts
- 65
I would setup a system snapshot that runs every minute or so and logs the current usage, processes, connections, etc... When this happens again you can view the logs to determine the exact cause.
-
04-13-2010, 09:10 AM #6Newbie
- Join Date
- Mar 2010
- Posts
- 13
how is the mysql service holding up
try mysqladmin proc
Similar Threads
-
apache hangs cpu 100% utilization many http requests
By eugene2006 in forum VPS HostingReplies: 8Last Post: 11-19-2006, 08:34 PM -
cpanel hangs after entering username and password
By juworld in forum Hosting Software and Control PanelsReplies: 2Last Post: 09-27-2005, 10:49 AM -
Server Hangs on Reboot - CentOS + CPanel
By VanHost in forum Hosting Security and TechnologyReplies: 5Last Post: 01-31-2005, 12:41 PM -
I try to upgrade cPanel, but it hangs..
By NOCTroll in forum Dedicated ServerReplies: 6Last Post: 12-25-2004, 05:50 AM -
Server hangs up
By singhk in forum Hosting Security and TechnologyReplies: 6Last Post: 04-26-2004, 08:15 AM