I know this is "late", but wanted to share anyway...
I've gotten this to work -- though on a new install, not an existing one. I just finished doing this on my MrNerd VPS, which is currently acting as a test VPS. You can create domains in VirtualMin, and control all the non-www services (mail, DNS, etc). Apache has to be disabled, unless you're going for a reverse proxy scenario, which I've not tried (and probably won't for now). The nginx vhosts has to be done in SSH. That VirtualMin nginx plugin doesn't work, neither version, so don't waste your time on it. Most htaccess rules can be converted to nginx rules.
It's not going to be easy (i.e., a few clicks), but it can be done if you really need to move to a better-than-Apache setup.
It will take you some time, so doing this on an existing production server will definitely result in downtime of at least a day, while you get everything migrated and tested. Take ample backups of the current VPS/dedicated box, in case you need to re-deploy off a bad migration. Test on a test VPS first, possibly using backups of your current server, and redirecting your HOSTS to the test location.
Can it be done? Sure.
Is it worth it? If given the choice of this or adding more RAM, I'd get more RAM. That's easier.