As long as you're not having it DO things all the time, you're good. It uses the majority of its RAM and CPU when you're accessing it then it tapers back off. I don't think the overhead is that significant if you're just using it from time to time.
Is virtualmin too much of a hog for these small systems, or is webmin just fine? Is webmin that much lighter than virtualmin?
Virtualmin will run fine just on 256MB Ram and actually uses a lot less but things like Bind, Apache, etc all take there far share of resources too.
The best thing about Virtualmin/Webmin is it's tight integration with Nginx as there is a module for it which is pretty neat makes administrating Nginx a piece of cake.