Yes, you can use your domain name and based your Name Servers off it.
It is a two step process.
First thing you need to do is to set up the name servers off your server. I am not sure which operating system or control panel you are using but you might want to consult the documentation for that on the specifics of setting it up.
Typically, you will need a dedicated IP address for each DNS. For instance, you could bind the following :-
You can using any hostname for the domain name, it is up to you. Once you have that setup.
Go to the Domain Name Registrar where you registered that domain name and select something like 'Create Nameserver based on the domain name'. It can be called Nameserver or DNS Server or Host. Make sure you correlate the IP address with the name server as defined in your server.
Wait for about 48 hours or so for the name server to populate.
To check if you done things right, telnet/SSH into your server and do the following test :-
to see if it internally resolved.
Then if you setup a domain name to point to the new Name servers, you could do a :-
nslookup mydomain.com ns1.yourname.com
This will check if the domain name entry is in the new name server.
Then if after the 48 hours is up, you could hop on to the external test arena.
to do all the testing if they resolve externally and also http://www.dnsreport.com
to see if the Name server entries for each domain name using it is set up properly.
An additional step would be to issue a support ticket to your web host and have them set up a reverse DNS for your DNS servers. Just tell them to set them up and provide the domain name and IP for the name servers.
Hope that helps.