-
Time_wait
Would an excess amount of these cause a server load issue? Past few days ive been having some issues with load and I was thinking it was a network issue. Scanning over the netstat Ive blocked some IPs via a hardware firewall (fortigate 300a) and load came down but new IPs have returned. Ive turned off keepalive in httpd.conf but now the TIME_WAIT has increased.
netstat -tan | grep ':80 ' | awk '{print $6}' | sort | uniq -c
1 ESTABLISHED
3 FIN_WAIT1
1 LISTEN
5 SYN_RECV
203 TIME_WAIT
Few mins later
netstat -tan | grep ':80 ' | awk '{print $6}' | sort | uniq -c
3 CLOSING
1 ESTABLISHED
1 LISTEN
4 SYN_RECV
118 TIME_WAIT
5 mins later
netstat -tan | grep ':80 ' | awk '{print $6}' | sort | uniq -c
2 CLOSING
1 LISTEN
1 SYN_RECV
174 TIME_WAIT
netstat search and sort top 10
18 69.110.35.171
15 66.249.66.12
10 61.90.235.89
6 70.51.210.91
4 128.194.135.81
3 82.43.187.252
2 81.38.10.78
2 72.30.226.144
2 68.142.251.38
2 68.142.250.95
With it on
netstat -tan | grep ':80 ' | awk '{print $6}' | sort | uniq -c
19 ESTABLISHED
5 FIN_WAIT2
1 LISTEN
3 SYN_RECV
69 TIME_WAIT
about 15 mins later
30 ESTABLISHED
1 LISTEN
2 SYN_RECV
54 TIME_WAIT
Kinda goes back and forth, with KeepAlive on that number is around 40-70
IPs are random, never the same so i dunno if im having some network problems or not kinda stumped. Just know the load is running a little rich for a robust system, procs are low, memory and swap is low, everything else looks alright. When load goes up a lot of httpd processes pop up but goes away just as quick as it showed up doesnt last long few seconds maybe but quick enough to raise the load to 12 and drops back down to 2, when they do thats all I see in netstat is a bunch of TIME_WAIT
This is now
netstat -tan | grep ':80 ' | awk '{print $6}' | sort | uniq -c 2 ESTABLISHED
1 LISTEN
2 SYN_RECV
312 TIME_WAIT
02:03:34 up 36 min, 2 users, load average: 4.20, 2.72, 1.63
131 processes: 125 sleeping, 5 running, 1 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 12.6% 0.0% 2.6% 0.1% 0.1% 6.9% 77.4%
cpu00 24.2% 0.0% 1.9% 0.1% 0.3% 0.3% 72.8%
cpu01 10.6% 0.0% 1.7% 0.0% 0.0% 0.0% 87.5%
cpu02 11.8% 0.0% 5.7% 0.3% 0.0% 26.8% 55.1%
cpu03 4.3% 0.0% 0.9% 0.0% 0.0% 0.5% 94.0%
Mem: 2074352k av, 1128268k used, 946084k free, 0k shrd, 85252k buff
831660k active, 237076k inactive
Swap: 4096532k av, 0k used, 4096532k free 578528k cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
9723 nobody 16 0 30692 17M 3224 S 2.3 0.8 0:08 2 httpd
9744 nobody 15 0 29616 15M 3284 S 2.3 0.7 0:02 0 httpd
9729 nobody 15 0 28920 15M 3064 S 2.1 0.7 0:04 0 httpd
9719 nobody 15 0 30992 17M 3200 R 1.9 0.8 0:13 3 httpd
12103 nobody 15 0 31736 18M 3240 S 1.4 0.8 0:08 0 httpd
12110 nobody 16 0 31500 17M 3256 R 1.0 0.8 0:04 3 httpd
9715 nobody 16 0 31956 18M 3416 R 0.8 0.9 0:11 1 httpd
6349 root 16 0 5292 1220 896 R 0.1 0.0 0:10 2 top
1 root 16 0 1592 496 436 S 0.0 0.0 0:01 0 init
2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
3 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
4 root RT 0 0 0 0 SW 0.0 0.0 0:00 1 migration/1
5 root 34 19 0 0 0 SWN 0.0 0.0 0:00 1 ksoftirqd/1
6 root RT 0 0 0 0 SW 0.0 0.0 0:00 2 migration/2
7 root 34 19 0 0 0 SWN 0.0 0.0 0:00 2 ksoftirqd/2
8 root RT 0 0 0 0 SW 0.0 0.0 0:00 3 migration/3
9 root 34 19 0 0 0 SWN 0.0 0.0 0:00 3 ksoftirqd/3
10 root 10 -5 0 0 0 SW< 0.0 0.0 0:00 0 events/0
11 root 10 -5 0 0 0 SW< 0.0 0.0 0:00 1 events/1
12 root 10 -5 0 0 0 SW< 0.0 0.0 0:00 2 events/2
13 root 10 -5 0 0 0 SW< 0.0 0.0 0:00 3 events/3
14 root 20 -5 0 0 0 SW< 0.0 0.0 0:00 3 khelper
15 root 10 -5 0 0 0 SW< 0.0 0.0 0:00 0 kthread
20 root 10 -5 0 0 0 SW< 0.0 0.0 0:00 0 kblockd/0
21 root 10 -5 0 0 0 SW< 0.0 0.0 0:00 1 kblockd/1
22 root 10 -5 0 0 0 SW< 0.0 0.0 0:00 2 kblockd/2
23 root 10 -5 0 0 0 SW< 0.0 0.0 0:00 3 kblockd/3
24 root 10 -5 0 0 0 SW< 0.0 0.0 0:00 0 kseriod
170 root 20 0 0 0 0 SW 0.0 0.0 0:00 1 pdflush
171 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 pdflush
172 root 16 0 0 0 0 SW 0.0 0.0 0:00 1 kswapd0
173 root 11 -5 0 0 0 SW< 0.0 0.0 0:00 0 aio/0
174 root 11 -5 0 0 0 SW< 0.0 0.0 0:00 1 aio/1
175 root 10 -5 0 0 0 SW< 0.0 0.0 0:00 2 aio/2
176 root 11 -5 0 0 0 SW< 0.0 0.0 0:00 3 aio/3
827 root 11 -5 0 0 0 SW< 0.0 0.0 0:00 0 scsi_eh_0