Memory and hard disk speed will be affected the most. Simple downloading doesn't require a superfast processor. Ofcourse to be able to give you specs, we need a rough estimate of server usage as datruesurfer suggested.
I have a 10Mbit unmetered Leaseweb box, Athlon 2700+, 40GB PATA, 512MB ram. It servers both dynamic webpages, and (static) big files.
At first I used only Apache and, well, that didn't run smooth. Apache used up way too much resources, slowing down everything.
So I decided to install thttpd right next too apache, only for serving the static big files. It had a huge effect: System load went down very nice.
One big disadvantage of thttpd though: It does not support resume. So if people use a download manager program, they cannot continue the download if it gets canceled or something like that.
So for that reason I moved from thttpd to a bare install of Lighttpd, which does support resume, and it runs great I'll move about 50GB per day, and system load is usually < 1. Sometimes it does go up a bit, but that is usually caused by a few page requests served by apache
As far as i know is Thttpd more suitable for serving huge amounts of small static files, like images and such. I prefer Lighttpd because it works better for me with huge files: Supporting Resume. (Which my visitors absolutely need).
I'm using :
P4 2.8 GHz HT, 1 GB Ram, 200 GB HDD ( now, it was fulled )
CentOS 3.4, CPanel/WHM, Apache.
* 20 Mbps unmetered BW
I have been using over 500 GB BW everyday
All of my files is over 300 MB
My server run very nice, server load always less than 1 , Ram used < 57%.
I really love this server
Your bottleneck will probably be HD's sata arent really that much faster than pata unless you have a hard drive and controller that support certain features. SCSI would be better but you might not want to spend that much. Your milage may vary (greatly) according to what kind of HD's are in that system.