hosted by liquidweb


Go Back   Web Hosting Talk : Web Hosting Main Forums : Programming Discussion : Perl or Java/Rails code to check for available domains
Reply

Forum Jump

Perl or Java/Rails code to check for available domains

Reply Post New Thread In Programming Discussion Subscription
 
Send news tip View All Posts Thread Tools Search this Thread Display Modes
  #1  
Old
Aspiring Evangelist
 
Join Date: Jul 2004
Posts: 375

Perl or Java/Rails code to check for available domains


I am looking for a script that can import a list of domains from a text file called domains.txt and check them one by one for availability. If a domain is available it will take the name and write it to a file called available.txt

This can be done using Perl (preferred) that I can run thru the command line or Rails (so I have heard).

I have read somewhere that you can use Perl or Rails to check for A records on the doamin. Most registered domains will have an A record so if it doesn't the script will consider this domain available. I can then take the names of available domains and conduct a more exact search for availability.

I would like to load in lists of domains that contain 5k to 15k of domains per list. It should be able to scan at a rate of 20-50 names per second.

Let me know if you know a script or if you can code it. PM me the price.



Sponsored Links
  #2  
Old
Web Hosting Master
 
Join Date: Nov 2001
Location: Vancouver
Posts: 2,416
I posted a Python script here sometime within the last 30 days or so that does almost exactly that. It was based off a script that I wrote for my own use that used a similar concept to find available domains. If there was no NS record (not A record) the domain was considered available.

The list of domains to check I created out of a large list of words and prefixes/suffixes in different combinations. I was getting 55 DNS results a second; I also cached the result of each run to a simple text format for persistence.

It wouldn't take much to run with the script I posted here:

http://www.webhostingtalk.com/showthread.php?t=615725

... and make it do what you want.

__________________
“Even those who arrange and design shrubberies are under
considerable economic stress at this period in history.”

  #3  
Old
Aspiring Evangelist
 
Join Date: Jul 2004
Posts: 375
I don't know Python, how much would you charge to customize it for me?

Sponsored Links
  #4  
Old
Newbie
 
Join Date: Apr 2003
Location: Charleston, SC
Posts: 28
jdk,

Here's a quick perl script that requires Net :: DNS (fairly common module).

Code:
#!/usr/bin/perl

# Domains Available
# Josh Skidmore <josh@vorcado.com>
# 05 August 2007 | 11:40p EST

# Requirements
	use Net::DNS;
	
# Variables
	%VAR	=	(
					db => './domains.txt',
				);
				
# Open file
	open (DB,$VAR{'db'});
	my (@domains) = <DB>;
	close (DB);

# Test domains
	foreach my $domain (@domains)
		{
			chomp($domain);
			
			my ($available) = &check_domain(domain => $domain);
			
			if ($available)
				{
					print "$domain is available.<br />\n";
				}
			else
				{
					print "$domain is NOT available<br />\n";
				}
		}
	
sub check_domain {
	
	# Test domain for existance
	# Josh Skidmore <josh@vorcado.com>
	# 05 August 2007 | 11:42p EST
	
	# Variables
		my (%DATA) = @_ ;
		my ($available) = 0;

	# Start Net::DNS
		my $res = Net::DNS::Resolver->new;
		$res->udp_timeout(2);
		$res->tcp_timeout(2);

		my ($domain) = $res->search($DATA{'domain'});
		
		if ($domain)
			{
				($available) = 1;
			}
			
	# Output
		return ($available);
}

__________________
Loookup.com - Find out information about any server, IP, or hostname

  #5  
Old
Web Hosting Master
 
Join Date: Aug 2005
Location: UK
Posts: 654
The fact that the domain name doesn't resolve does not mean it's available.

You should use a script that checks with a registrar. I use the enom API to check domain name availability, but all the larger resellers have these too.

  #6  
Old
Newbie
 
Join Date: Apr 2006
Posts: 12
I would suggest Perl. And use some 3rd-party whois service.

  #7  
Old
Web Hosting Master
 
Join Date: Nov 2001
Location: Vancouver
Posts: 2,416
Quote:
Originally Posted by sibsoft View Post
I would suggest Perl.
Why? Does it matter which language such a utility is written in?

__________________
“Even those who arrange and design shrubberies are under
considerable economic stress at this period in history.”

  #8  
Old
Web Hosting Master
 
Join Date: Nov 2001
Location: Vancouver
Posts: 2,416
Code:
% ./domain_checker.py 
Domains to process: 566
Available domains written to file: 420
Run time: 1.32230615616
I decided that 25 - 50 results per second was too slow so I incorporated the example given into a threaded version which follows in this post. Depending on the state of your name server's cache, between 5 and 10 workers will probably do the job but regardless, its processing over 400 domains per second on a somewhat old desktop of mine. That ought to be fast enough.

Taking the example code of mine that I linked to earlier, you can see that the core code for checking the domains is only 3 lines of name server checking code plus a few lines here and there to read candidates from a test file and then write them back. Writing the python code to do the actual work took all of a few minutes. Adding in option parsing and threading took a little more time, mostly because I don't write threaded code very often. But still measured in minutes, not hours.

No charge.

In order to run this code you'll need python installed of course, plus download and install py-dns from http://pydns.sourceforge.net. Installing most python packages is dirt simple - usually involves no more than:

Code:
python setup.py install
Have at it.

(used PHP hilighter just to make this more readable on-line; it is Python of course)

PHP Code:
#!/usr/bin/env python
"""
$URL$
$Id$

A multi-threaded DNS query solution, originally developed to check a
large list of candidate domain names -- if no NS is present, the candidate
likely does not have a domain registration.

(C) mwatkins, a WebHostingTalk.com user. Use for any purpose you like provided
attribution is provided.
"""
# install PyDNS: http://pydns.sourceforge.net
# FreeBSD: /usr/ports/dns/py-dns

import DNS
import threading
import time
import Queue
from optparse import OptionParser

DNS
.ParseResolvConf()

WORKERS 10

def has_ns 
(domain):
    
"""(domain:string) -> boolean
    """
    
query DNS.Request(domainqtype="NS").req()
    
status query.header['status']
    return (
domainstatus)


def get_domains_from_file(path):
    
"""(path:string) -> [string]

    Returns domains from a text file as a list of strings.
    """
    
return [domain.strip() for domain in open(path).readlines()
            if 
domain.strip()]


class 
Worker(threading.Thread):

    
def __init__(selfwork_queueresult_queue):
        
self.__queue work_queue
        self
.__result result_queue
        threading
.Thread.__init__(self)

    
def run(self):
        while 
1:
            
domain self.__queue.get()
            if 
domain is None:
                break
            
self.__result.put(has_ns(domain))


def process_domains(domains):
    
"""(domains:sequence) -> ([string], [string])

    Processes a sequence of domain strings and returns two lists
    containing those that are potentially available for registration,
    and those which have name servers defined and are therefore unavailable.

    A final check for availability against a registrar whois API is recommended.
    """
    
work Queue.Queue()
    
result Queue.Queue()
    for 
domain in domains:
        
work.put(domain)
    
# add work terminators
    
for i in range(WORKERS):
        
work.put(None)

    
workers = [Worker(workresult) for i in range(WORKERS)]
    for 
worker in workers:
        
worker.start()

    
# wait for workers to complete
    
for w in workers:
        
w.join()

    
available = []
    
unavailable = []
    while 
not result.empty():
        
domainstatus result.get(0)
        if 
status == 'NXDOMAIN':
            
available.append(domain)
        else:
            
unavailable.append(domain)

    return 
availableunavailable


def run_domain_checker_main
():
    global 
WORKERS

    parser 
OptionParser()
    
parser.set_description('Checks domain names for available name servers. '
        'Domain names with no name servers defined are potentially available '
        'for registration. A final check for availability against a registrar '
        'whois API is recommended.'
)
    
parser.add_option(
        
'-f''--inputfile'dest='input_file', default='domains.txt',
        
help='A file containing one domain name per line ')
    
parser.add_option(
        
'-o''--outputfile'dest='output_file', default='available.txt',
        
help=('A file to contain the candidate domains which may '
              'be available. Note: This file is overwritten on every run.'
))
    
parser.add_option(
        
'-s''--server'dest='dns_server', default=None,
        
help='Force use of a specific name server, overriding /etc/resolve.conf.')
    
parser.add_option(
        
'-t''--threads'dest='workers', default=10type=int,
        
help='Number of worker threads')
    
parser.add_option(
        
'-c''--cache'action="store_true"dest='use_cache', default=False,
        
help=('Use contents of <outputfile>, if it exists, as a cache. '
              'In other words, do not requery domains that have been found '
              'to be available. Newly found "available" domains will be '
              'added to the cache in <outputfile>.'
))

    
optionsargs parser.parse_args()
    
domains get_domains_from_file(options.input_file)
    
cached_domains = []
    
WORKERS options.workers
    
if options.dns_server:
        
DNS.defaults['server'] = [options.dns_server,]
    if 
options.use_cache:
        try:
            
cached_domains get_domains_from_file(options.output_file)
        
except IOErrore:
            print 
"INFO: Skipping cache"e
        
# we don't want to check those that already are in the output file
        
domains = [domain for domain in domains if domain not in
                   cached_domains
]
    
start time.time()
    print 
"Domains to process:"len(domains)
    
availunavail process_domains(domains)
    
fout open(options.output_file'w')
    
# we'll add cached_domains back to the output ([] if the option wasn't specified)
    
result sorted(avail cached_domains)
    
fout.writelines(['%s\n' domain for domain in result])
    
fout.close()
    print 
"Available domains written to file:"len(result)
    print 
"Run time:"time.time() - start

if __name__ == '__main__':
    
run_domain_checker_main() 

__________________
“Even those who arrange and design shrubberies are under
considerable economic stress at this period in history.”

Reply

Related posts from TheWhir.com
Title Type Date Posted
Web Hosting Sales and Promos Roundup - September 26, 2014 Web Hosting News 2014-09-26 15:37:14
Web Hosting Sales and Promos Roundup – August 2, 2013 Web Hosting News 2014-05-23 15:42:54
Cloud Platform Heroku Offers Free Ruby on Rails Training for Developers Web Hosting News 2013-04-12 14:41:30
Ruby on Rails Vulnerabilities Discovered, Users Urged to Update Immediately Web Hosting News 2013-01-10 11:05:11
cPanel Security Updates Address Perl Module Vulnerabilities Web Hosting News 2012-12-06 12:55:54


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes
Postbit Selector

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump
Login:
Log in with your username and password
Username:
Password:



Forgot Password?
WHT Host Brief Email:

We respect your privacy. We will never sell, rent, or give away your address to any outside party, ever.

Advertisement:
Web Hosting News:
WHT Membership
WHT Membership



 

X

Welcome to WebHostingTalk.com

Create your username to jump into the discussion!

WebHostingTalk.com is the largest, most influentual web hosting community on the Internet. Join us by filling in the form below.


(4 digit year)

Already a member?