how do I change the max number of mailboxes under Ensim 3?

You need to be signed in as Appliance Admin, pull up the domain you want to change and alter the Max Users setting. This limits the number of POP3 box 'users'.