If you are using cPanel I would reccomend 768MB/1GB RAM, with some optimization it will be fine with that traffic. You can get away with less Ram on DirectAdmin as its a much simpler and lighter panel.
Of course for the best performance no panel and Lighttpd would offer the best performance.
Yes, swap will not run a web site, and if it does it would be a very slow one ;-).
SWAP is there to guarantee your VPS won't run out of memory when you hit the limit but it isn't their to rely on for normal usage.
I'd definitely suggest a 1GB VPS for the kind of traffic you are expecting to receive.
I always suggest DirectAdmin over CPanel because it is lighter and in my opinion just easier to find your way around, although you won't have access to any HTTP interface for apache/php updates like WHM but if you have a managed VPS then that isn't anything to worry about.