Err.... I think I know what you are asking.
For example, if I own the domain abc.com, it can reside on ns1.xyz.com, ns2.xyz.com etc. Basically, ns1.xyz.com has the "Start of Authority (SOA)" for the abc.com domain, and can therefore resolve queries from
http://www.abc.com to the physical IP address of the box.
If doesn't matter that name server you specify, as long as:
1) You're registratar is told which DNS has the SOA of the domain.
2) The people running the name server know that they are supposed to have your domain list in it.
If you were to go to a hosting company, your domain usually gets listed in the name server, and you enter ns1.myhostingcompany.com as the name server at your registrar. Alternatively, you can run your own name servers, and resolve your own domain and others if so desired.
One final example, my t595.net SOA resides at ns1.mydomain.com - ns4.mydomain.com, which means that mydomain.com owns the task of resolving queries to my t595.net domain.
err... just re-read your question, misinterpreted. It is best to have your domain removed from the old name server when you move, mainly in the interest of cleanliness (and because some hosting companies set up their DNS incorrectly). In theory, only the name server with the SOA as specified by your registrar will be queried for your domain. However, some hosts use the name servers as both caching and resolution servers. This means that their name servers still think they own the SOA, and will resolve queries. However, queries will not normally reach them, but because they are running the name server as a caching server for the local network, it get resolved incorrectly. The biggest bugbear for this is when signing up to a new host, and they try to send a confirmation email - the email will go to your new email server at the new host, instead of going to your current server, so you never get your email with details on the new account.