Assuming it's a Linux box, then adding more RAM to the file server will roughly achieve the same thing, as Linux will cache areas of the file system in RAM for faster access when they are needed again.
You would put a squid "reverse proxy" box in front of your web server.
I would recommend doing what the others have said first. Max out the ram on the fileserver (upgrade to 64bit if it is not already) and you will be happy. Ram is stupid cheap these days. Under $150 for 4gb (2x2gb) of server ram from newegg. Most server boards have 8-16 slots for ram. 16gb-32gb is easily possible.
The fileserver in question have some big slow sata2 drives in it. And can get quite a bit slow when it's getting hit a lot.
So what I would do with the "memcache-box" is to stick 4 15k rpm SAS drives in it and fill it up with as much ram as possible. 2 SAS drives for os, and 2 for swap. The box would then serve the most popular files. This would considerably lower the amount of connections and IO on the fileserver.
My idea was also using a reverse squid proxy or something similar
I use memcache as an example cos it would describe what the use of the server would be.
Squid performance tends to be great right until the point that it isn't anymore. What I mean by that is up to a certain threshold it will perform well, but when you hit the wall it will be very noticeable.
That sounds like something that would happen when the cache moves from memory to swap. Thanks for the tip though. Would need to benchmark it and monitor it for a while. Might even have to load balance 2 squid boxes