DoS protection has limits, and DDoS is near impossible to handle unless you are using hardware... even then it is a battle for resources.
You may want to consider the following on a UNIX-box:
* Disable telnet.
* Limit SSH access to specific IP addresses.
* Disable direct root login.
* Remove unnecessary packages / software.
* Harden the kernel against synflood and basic DOS attacks.
* Remove common user access to compilers and fetching software (wget, fetch, lynx, etc.).
* Ensure /tmp is in its own partition with noexec, nosuid.
* Ensure kernel and software is up to date.
* Remove unnecessary users and groups.
* Install chkrootkit, logwatch, tripwire.
* Install a firewall, and port scan detector.
* For Apache servers, install mod_security and configure for use with FrontPage, PHPMyAdmin, Site Studio, and common applications.
* Secure DNS Servers
* Utilize firewall automation to mitigate brute force FTP, syn floods, mail bombs, and out-of-network trojan’d servers from impacting your servers
It is important to note that security is an ongoing venture. Even if you were to take all of the steps listed above, you would still have a regular routine of review, update, research, patch, etc.