Hi,

This is the first time I'm having to actually ask for help here since I've been searching the internet for a solution all day and have drawn a blank so far..

First off, system info:
CentOS 5.3
Dual Xeon 3.0Ghz
4Gb Ram
2x1TB Sata Drives
300Mbit Capped Gige Port

This system is to be used as an ftp server.
So I setup partitions, moved a ton of data over. Installed vsftpd. Ran it. Users started downloading, BOOM! High IOWait, High Server Load.. =/

Everytime vsftpd gets about 700 concurrent users, the server load starts skyrocketing (mainly because of the high iowait 80-90%..). I thought it might be a disk issue, so I ran:
Code:
hdparm -Tt /dev/sda1
hdparm -Tt /dev/sdb1
Both gave me good results [speeds of 100+MB/s]

I thought it might be a kernel issue, so I tried these kernels:

CentOS (2.6.18-128.7.1.el5)
CentOS (2.6.18-164.el5)
CentOS (2.6.18-164.el5PAE)
CentOS (2.6.18-128.el5PAE)

No joy

All seem to be giving the same results..

Right now, there're 167 vsftpd processes:
Code:
top - 17:16:01 up 9 min,  1 user,  load average: 0.34, 0.21, 0.09
Tasks: 262 total,   1 running, 260 sleeping,   0 stopped,   1 zombie
Cpu(s):  0.2%us,  0.7%sy,  0.0%ni, 94.1%id,  3.7%wa,  0.2%hi,  1.0%si,  0.0%st
Mem:   3631768k total,  3503940k used,   127828k free,    11304k buffers
Swap:  8193140k total,        0k used,  8193140k free,  3420304k cached
Everything seems to be running beautifully. But once the number of users go up.. as they will in the next 15 minutes, things slow down to a crawl.

[root@ID5961 vs]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 895G 647G 202G 77% /
/dev/sda1 289M 32M 243M 12% /boot
/dev/sdb1 917G 818G 53G 94% /mnt/new
tmpfs 1.8G 0 1.8G 0% /dev/shm

I'm not sure what's causing the bottleneck here. If hdparm is giving good results, what is causing the IO bottleneck ?

Please help!


EDIT: Oh, by the way, I tried changing the IO scheduler as well [it's cfq by default..], tried every one of them. Worse results on every other.

Oh yes, and just incase:
[root@ID5961 vs]# uname -a
Linux ID5961.choopa.com 2.6.18-128.7.1.el5 #1 SMP Mon Aug 24 08:20:55 EDT 2009 i686 i686 i386 GNU/Linux