for OS I'd recommend linux - it's a bit safer in terms of virus effects
as for hardware, you'd have to look at how much emails will be sent out and received. If it's just a few emails an hour, then you could get away with a low end server with a single core processor and just 1gb of ram. But if you plan on sending/receiving thousands of emails every hour, then you'll need something stronger since that many emails can put a good amount of load on the server.
Linux definitely. As for hardware specs a multi core processor will be good but you still need to have more information on the needs of server resources to allow your applications to run effectively. Also when looking for a provider the number of emails and how many you intend to send out might be a bit of an issue. Most companies will require that you have an opt out feature if you intend to send mass emails.
Miguel Villegas, VP Sales & New Business Development CloudMega.net
CloudMega, Inc. US | CloudMega, Ltd UK
US Toll Free: 888.401.2006 | Intl +1 312.757.5400
UK Toll Free: 0800.368.9925 | Intl +44 20.3540.7807
As they mentioned above ..the intensity of email will determine you hardware (majority of CPU time is consumed on the scanning part more the MX part) ..so it better to overbuild than under build in this case ..get the best your budget can afford you.