Results 1 to 8 of 8
  1. #1
    Join Date
    Jul 2004
    Posts
    379

    Perl or Java/Rails code to check for available domains

    I am looking for a script that can import a list of domains from a text file called domains.txt and check them one by one for availability. If a domain is available it will take the name and write it to a file called available.txt

    This can be done using Perl (preferred) that I can run thru the command line or Rails (so I have heard).

    I have read somewhere that you can use Perl or Rails to check for A records on the doamin. Most registered domains will have an A record so if it doesn't the script will consider this domain available. I can then take the names of available domains and conduct a more exact search for availability.

    I would like to load in lists of domains that contain 5k to 15k of domains per list. It should be able to scan at a rate of 20-50 names per second.

    Let me know if you know a script or if you can code it. PM me the price.

  2. #2
    Join Date
    Nov 2001
    Location
    Vancouver
    Posts
    2,422
    I posted a Python script here sometime within the last 30 days or so that does almost exactly that. It was based off a script that I wrote for my own use that used a similar concept to find available domains. If there was no NS record (not A record) the domain was considered available.

    The list of domains to check I created out of a large list of words and prefixes/suffixes in different combinations. I was getting 55 DNS results a second; I also cached the result of each run to a simple text format for persistence.

    It wouldn't take much to run with the script I posted here:

    http://www.webhostingtalk.com/showthread.php?t=615725

    ... and make it do what you want.
    “Even those who arrange and design shrubberies are under
    considerable economic stress at this period in history.”

  3. #3
    Join Date
    Jul 2004
    Posts
    379
    I don't know Python, how much would you charge to customize it for me?

  4. #4
    Join Date
    Apr 2003
    Location
    Charleston, SC
    Posts
    28
    jdk,

    Here's a quick perl script that requires Net :: DNS (fairly common module).

    Code:
    #!/usr/bin/perl
    
    # Domains Available
    # Josh Skidmore <josh@vorcado.com>
    # 05 August 2007 | 11:40p EST
    
    # Requirements
    	use Net::DNS;
    	
    # Variables
    	%VAR	=	(
    					db => './domains.txt',
    				);
    				
    # Open file
    	open (DB,$VAR{'db'});
    	my (@domains) = <DB>;
    	close (DB);
    
    # Test domains
    	foreach my $domain (@domains)
    		{
    			chomp($domain);
    			
    			my ($available) = &check_domain(domain => $domain);
    			
    			if ($available)
    				{
    					print "$domain is available.<br />\n";
    				}
    			else
    				{
    					print "$domain is NOT available<br />\n";
    				}
    		}
    	
    sub check_domain {
    	
    	# Test domain for existance
    	# Josh Skidmore <josh@vorcado.com>
    	# 05 August 2007 | 11:42p EST
    	
    	# Variables
    		my (%DATA) = @_ ;
    		my ($available) = 0;
    
    	# Start Net::DNS
    		my $res = Net::DNS::Resolver->new;
    		$res->udp_timeout(2);
    		$res->tcp_timeout(2);
    
    		my ($domain) = $res->search($DATA{'domain'});
    		
    		if ($domain)
    			{
    				($available) = 1;
    			}
    			
    	# Output
    		return ($available);
    }
    Loookup.com - Find out information about any server, IP, or hostname

  5. #5
    Join Date
    Aug 2005
    Location
    UK
    Posts
    654
    The fact that the domain name doesn't resolve does not mean it's available.

    You should use a script that checks with a registrar. I use the enom API to check domain name availability, but all the larger resellers have these too.

  6. #6
    I would suggest Perl. And use some 3rd-party whois service.

  7. #7
    Join Date
    Nov 2001
    Location
    Vancouver
    Posts
    2,422
    Quote Originally Posted by sibsoft View Post
    I would suggest Perl.
    Why? Does it matter which language such a utility is written in?
    “Even those who arrange and design shrubberies are under
    considerable economic stress at this period in history.”

  8. #8
    Join Date
    Nov 2001
    Location
    Vancouver
    Posts
    2,422
    Code:
    % ./domain_checker.py 
    Domains to process: 566
    Available domains written to file: 420
    Run time: 1.32230615616
    I decided that 25 - 50 results per second was too slow so I incorporated the example given into a threaded version which follows in this post. Depending on the state of your name server's cache, between 5 and 10 workers will probably do the job but regardless, its processing over 400 domains per second on a somewhat old desktop of mine. That ought to be fast enough.

    Taking the example code of mine that I linked to earlier, you can see that the core code for checking the domains is only 3 lines of name server checking code plus a few lines here and there to read candidates from a test file and then write them back. Writing the python code to do the actual work took all of a few minutes. Adding in option parsing and threading took a little more time, mostly because I don't write threaded code very often. But still measured in minutes, not hours.

    No charge.

    In order to run this code you'll need python installed of course, plus download and install py-dns from http://pydns.sourceforge.net. Installing most python packages is dirt simple - usually involves no more than:

    Code:
    python setup.py install
    Have at it.

    (used PHP hilighter just to make this more readable on-line; it is Python of course)

    PHP Code:
    #!/usr/bin/env python
    """
    $URL$
    $Id$

    A multi-threaded DNS query solution, originally developed to check a
    large list of candidate domain names -- if no NS is present, the candidate
    likely does not have a domain registration.

    (C) mwatkins, a WebHostingTalk.com user. Use for any purpose you like provided
    attribution is provided.
    """
    # install PyDNS: http://pydns.sourceforge.net
    # FreeBSD: /usr/ports/dns/py-dns

    import DNS
    import threading
    import time
    import Queue
    from optparse import OptionParser

    DNS
    .ParseResolvConf()

    WORKERS 10

    def has_ns 
    (domain):
        
    """(domain:string) -> boolean
        """
        
    query DNS.Request(domainqtype="NS").req()
        
    status query.header['status']
        return (
    domainstatus)


    def get_domains_from_file(path):
        
    """(path:string) -> [string]

        Returns domains from a text file as a list of strings.
        """
        
    return [domain.strip() for domain in open(path).readlines()
                if 
    domain.strip()]


    class 
    Worker(threading.Thread):

        
    def __init__(selfwork_queueresult_queue):
            
    self.__queue work_queue
            self
    .__result result_queue
            threading
    .Thread.__init__(self)

        
    def run(self):
            while 
    1:
                
    domain self.__queue.get()
                if 
    domain is None:
                    break
                
    self.__result.put(has_ns(domain))


    def process_domains(domains):
        
    """(domains:sequence) -> ([string], [string])

        Processes a sequence of domain strings and returns two lists
        containing those that are potentially available for registration,
        and those which have name servers defined and are therefore unavailable.

        A final check for availability against a registrar whois API is recommended.
        """
        
    work Queue.Queue()
        
    result Queue.Queue()
        for 
    domain in domains:
            
    work.put(domain)
        
    # add work terminators
        
    for i in range(WORKERS):
            
    work.put(None)

        
    workers = [Worker(workresult) for i in range(WORKERS)]
        for 
    worker in workers:
            
    worker.start()

        
    # wait for workers to complete
        
    for w in workers:
            
    w.join()

        
    available = []
        
    unavailable = []
        while 
    not result.empty():
            
    domainstatus result.get(0)
            if 
    status == 'NXDOMAIN':
                
    available.append(domain)
            else:
                
    unavailable.append(domain)

        return 
    availableunavailable


    def run_domain_checker_main
    ():
        global 
    WORKERS

        parser 
    OptionParser()
        
    parser.set_description('Checks domain names for available name servers. '
            'Domain names with no name servers defined are potentially available '
            'for registration. A final check for availability against a registrar '
            'whois API is recommended.'
    )
        
    parser.add_option(
            
    '-f''--inputfile'dest='input_file', default='domains.txt',
            
    help='A file containing one domain name per line ')
        
    parser.add_option(
            
    '-o''--outputfile'dest='output_file', default='available.txt',
            
    help=('A file to contain the candidate domains which may '
                  'be available. Note: This file is overwritten on every run.'
    ))
        
    parser.add_option(
            
    '-s''--server'dest='dns_server', default=None,
            
    help='Force use of a specific name server, overriding /etc/resolve.conf.')
        
    parser.add_option(
            
    '-t''--threads'dest='workers', default=10type=int,
            
    help='Number of worker threads')
        
    parser.add_option(
            
    '-c''--cache'action="store_true"dest='use_cache', default=False,
            
    help=('Use contents of <outputfile>, if it exists, as a cache. '
                  'In other words, do not requery domains that have been found '
                  'to be available. Newly found "available" domains will be '
                  'added to the cache in <outputfile>.'
    ))

        
    optionsargs parser.parse_args()
        
    domains get_domains_from_file(options.input_file)
        
    cached_domains = []
        
    WORKERS options.workers
        
    if options.dns_server:
            
    DNS.defaults['server'] = [options.dns_server,]
        if 
    options.use_cache:
            try:
                
    cached_domains get_domains_from_file(options.output_file)
            
    except IOErrore:
                print 
    "INFO: Skipping cache"e
            
    # we don't want to check those that already are in the output file
            
    domains = [domain for domain in domains if domain not in
                       cached_domains
    ]
        
    start time.time()
        print 
    "Domains to process:"len(domains)
        
    availunavail process_domains(domains)
        
    fout open(options.output_file'w')
        
    # we'll add cached_domains back to the output ([] if the option wasn't specified)
        
    result sorted(avail cached_domains)
        
    fout.writelines(['%s\n' domain for domain in result])
        
    fout.close()
        print 
    "Available domains written to file:"len(result)
        print 
    "Run time:"time.time() - start

    if __name__ == '__main__':
        
    run_domain_checker_main() 
    “Even those who arrange and design shrubberies are under
    considerable economic stress at this period in history.”

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •