If you have a common name server, if that server is down, all the server becomes useless. Even if they are not used or they are used very less, it is important to segregate your nameservers according to business models.
I feel it is right at this moment, apart from only hassle that, when you want to edit any entries, you need to remember which server it has to goto.
And for smaller loads, having common name server does not simplify your job at all, if you have S1 as everyone's dns and you have S2,S3,S4 without dns. If your S1 goes down, which is DNS, your all S2,S3,S4 becomes useless because no one will be able to connect to (there will be chances that people will still will be able to connect to S2,S3,S4 using their cached dns entries of ISP), but you wil not get new visitors. If your S2,S3,S4 or any of them are down, its no point having your DNS running on S1.
Unless you have huge loads and huge entries like 500+ entries on dns server, or you require same website to have mirrors on many servers, your setup i feel is correct.