Does anyone know of any brute force prevention software?
Preferrably one that can protect a whole dedicated server not just a directory.

I'm quite suprised to see the lack of web hosts out there without any brute force prevention methods. I don't think any customer would object to having a certain IP address blocked if a hacker was running a brute force attack.

You could achieve this, by simply restricting the number of tries each ip can have at once. for example a maximum of 3 continuesly and the others have to be after 5mins.

Really? Where is this setting?

Honestly, you are better off doing this kind of prevention at the router or switch level, using ACLs, and not at the server level.

If you do descide to implement ACL-type security at the kerlel level, there is a really good how-to about IPSec at:

http://www.linuxdoc.org/LDP/lasg/ipsec/index.html