You can register the nameservers at your domain registrar for that domain. You would also then need to make sure your VPS will answer to those nameservers. This is really easy in cPanel. Are you using cPanel?
I have my main site on one server, my billing system on its own server, and my customers on seperate servers. Now I use advance dns systems. I only use the ns records for my customers. i point my a record @ to my main site and another a record to my biling system and i have name servers setup with the ip address of my servers for my clients.
I will probably soon just have a dns server. I like it because I can keep the dns settings on a seperate service like cloudflare which I will be moving to soon. and not have to worry that if my main site goes down and my dns settings are on that server I wont have to worry about changing the name servers and waiting for them to propogate. I just change the ip in the a record or add a second one to a secondary backup server. I have used it for a while.
I actually use a seperate domain for my clients and servers than my main site. It makes it easier for me to manage the dns settings as my main site domain was registered at godaddy and I am not a fan of their service I just havent moved the domain to a different company or to where my "server domain is" its kind of like how hostgator and other companies do where they have hostgator.com but have websitewelcome.com for their shared and reseller servers I have the same concept.
If you are on a cPanel server, then create your private nameserver by creating A record like:
ns1.domain.com and ns2.domain.com
Now login in your domain control panel and create child nameserver with same IP.
█ YagHost - Pure SSD Hosting | Since 2007 | Average Response Time: 15 min
█ Web Hosting | Reseller Hosting | Managed VPS Hosting
█ 99.9% Server Uptime Guarantee | 24/7 Rapid Response Tech Support | 30 Day Money Back Guarantee
█ LopHost.com - Web Hosting Tutorials