I'm running a server daemon that over 1000 people connect to at the same time and that worked perfectly when running on FreeBSD 5.4. However, when I migrated to CentOS 4, the daemon started outputting "accept: Bad file descriptor" errors on the console when it hit 1024 open file descriptors and disconnecting the users trying to connect. ulimit -n quickly revealed that the open file limit was set to 1024 and responsible for this. I read into how to raise the limit in CentOS and tried the following things (in different combinations):
- Making sure /proc/sys/fs/file-max was big enough
- Adding "fs.file-max = 2048" to /etc/sysctl.conf
- Adding "* - nofile 2048" to /etc/security/limits.conf
- Adding "session required pam_limits.so" to /etc/pam.d/login and /etc/pam.d/sshd
- Changing the limit using "ulimit -n 65536" as root
- Restarting sshd several times
- Rebooting the machine several times
None of this seemed to work.. in fact at some point the daemon would actually segment fault when it hit the limit instead of disconnecting users attempting to connect... Lowering the limit to 1024 in /etc/security/limits.conf stopped that and brought me back to the bad file descriptor errors. Once again, the same software worked just fine with over 1024 users on FreeBSD 5.4, so I'm pretty sure the crashing isn't the software's fault.
Am I overlooking something in the CentOS config that would cause this? I'm out of ideas.. and I've tried just about anything google would come up with.