I know this is "late", but wanted to share anyway...
I've gotten this to work -- though on a new install, not an existing one. I just finished doing this on my MrNerd VPS, which is currently acting as a test VPS. You can create domains in VirtualMin, and control all the non-www services (mail, DNS, etc). Apache has to be disabled, unless you're going for a reverse proxy scenario, which I've not tried (and probably won't for now). The nginx vhosts has to be done in SSH. That VirtualMin nginx plugin doesn't work, neither version, so don't waste your time on it. Most htaccess rules can be converted to nginx rules.
It's not going to be easy (i.e., a few clicks), but it can be done if you really need to move to a better-than-Apache setup.
It will take you some time, so doing this on an existing production server will definitely result in downtime of at least a day, while you get everything migrated and tested. Take ample backups of the current VPS/dedicated box, in case you need to re-deploy off a bad migration. Test on a test VPS first, possibly using backups of your current server, and redirecting your HOSTS to the test location.
Can it be done? Sure.
Is it worth it? If given the choice of this or adding more RAM, I'd get more RAM. That's easier.
Virtualmin has rolled out their NginX web server module. It is working well. I have it in public test on a VPS from Semoweb.
search for info on WordPress with W3 Total Cache on Nginx with APC on Virtualmin and you will find a blog post regarding the setup I used, and the proper nginx config and includes for Wordpress pretty urls and w3 total cache.
There is also some info on the Virtualmin site regarding how to switch from Apache to NginX, though it's recommended to start with a fresh install and create your virtual domains with nginx from the ground up. But it is possible to switch from Apache to NginX, because the directory hierarchy is standard virtualmin, /home/domain.com/public_html/...