Visnetic MailServer with WebMail (deerfield.com) will allow new users to sign up. It is very very simple. It will NOT, however, allow the users pay and keep track of all that sort of thing. It just allows the user to sign-up, instantly gives them an email account and that's it. You would have to integrate your own sort of payment system.
Remeber though, once installed, you have to log in the web interface as the admin account and enable the signup process and set the default domain (if you have more than one) to let the user sign up under. I think you can allow them to select from a list of domains too.
for what it's worth, once installed, self.html is the file that lets the user register. you can customize many aspects of how all that works with your own html code and scripts.