If it is for video clips, encoding it in flash is a great way to go.
Easy to download, no worries.
Certainly dont need much in the way of processor speed.
As far as server specs, it will depend on quantity of concurrent downloads. If you are only going to have a few people downloading simultaneously, then you can get by with a simple hard drive and minimum quantity of ram.
If you are going to be experiencing hundreds of simultaneous downloads, then you will want multiple hard drives, RAID, and extra ram.