Results 1 to 8 of 8
-
08-05-2007, 04:07 PM #1Aspiring Evangelist
- Join Date
- Jul 2004
- Posts
- 379
Perl or Java/Rails code to check for available domains
I am looking for a script that can import a list of domains from a text file called domains.txt and check them one by one for availability. If a domain is available it will take the name and write it to a file called available.txt
This can be done using Perl (preferred) that I can run thru the command line or Rails (so I have heard).
I have read somewhere that you can use Perl or Rails to check for A records on the doamin. Most registered domains will have an A record so if it doesn't the script will consider this domain available. I can then take the names of available domains and conduct a more exact search for availability.
I would like to load in lists of domains that contain 5k to 15k of domains per list. It should be able to scan at a rate of 20-50 names per second.
Let me know if you know a script or if you can code it. PM me the price.
-
08-05-2007, 04:38 PM #2Web Hosting Master
- Join Date
- Nov 2001
- Location
- Vancouver
- Posts
- 2,422
I posted a Python script here sometime within the last 30 days or so that does almost exactly that. It was based off a script that I wrote for my own use that used a similar concept to find available domains. If there was no NS record (not A record) the domain was considered available.
The list of domains to check I created out of a large list of words and prefixes/suffixes in different combinations. I was getting 55 DNS results a second; I also cached the result of each run to a simple text format for persistence.
It wouldn't take much to run with the script I posted here:
http://www.webhostingtalk.com/showthread.php?t=615725
... and make it do what you want.“Even those who arrange and design shrubberies are under
considerable economic stress at this period in history.”
-
08-05-2007, 05:34 PM #3Aspiring Evangelist
- Join Date
- Jul 2004
- Posts
- 379
I don't know Python, how much would you charge to customize it for me?
-
08-05-2007, 11:51 PM #4Newbie
- Join Date
- Apr 2003
- Location
- Charleston, SC
- Posts
- 28
jdk,
Here's a quick perl script that requires Net :: DNS (fairly common module).
Code:#!/usr/bin/perl # Domains Available # Josh Skidmore <josh@vorcado.com> # 05 August 2007 | 11:40p EST # Requirements use Net::DNS; # Variables %VAR = ( db => './domains.txt', ); # Open file open (DB,$VAR{'db'}); my (@domains) = <DB>; close (DB); # Test domains foreach my $domain (@domains) { chomp($domain); my ($available) = &check_domain(domain => $domain); if ($available) { print "$domain is available.<br />\n"; } else { print "$domain is NOT available<br />\n"; } } sub check_domain { # Test domain for existance # Josh Skidmore <josh@vorcado.com> # 05 August 2007 | 11:42p EST # Variables my (%DATA) = @_ ; my ($available) = 0; # Start Net::DNS my $res = Net::DNS::Resolver->new; $res->udp_timeout(2); $res->tcp_timeout(2); my ($domain) = $res->search($DATA{'domain'}); if ($domain) { ($available) = 1; } # Output return ($available); }
Loookup.com - Find out information about any server, IP, or hostname
-
08-06-2007, 12:30 AM #5Web Hosting Master
- Join Date
- Aug 2005
- Location
- UK
- Posts
- 654
The fact that the domain name doesn't resolve does not mean it's available.
You should use a script that checks with a registrar. I use the enom API to check domain name availability, but all the larger resellers have these too.
-
08-07-2007, 06:33 AM #6Newbie
- Join Date
- Apr 2006
- Posts
- 15
I would suggest Perl. And use some 3rd-party whois service.
-
08-07-2007, 04:47 PM #7Web Hosting Master
- Join Date
- Nov 2001
- Location
- Vancouver
- Posts
- 2,422
-
08-07-2007, 05:15 PM #8Web Hosting Master
- Join Date
- Nov 2001
- Location
- Vancouver
- Posts
- 2,422
Code:% ./domain_checker.py Domains to process: 566 Available domains written to file: 420 Run time: 1.32230615616
Taking the example code of mine that I linked to earlier, you can see that the core code for checking the domains is only 3 lines of name server checking code plus a few lines here and there to read candidates from a test file and then write them back. Writing the python code to do the actual work took all of a few minutes. Adding in option parsing and threading took a little more time, mostly because I don't write threaded code very often. But still measured in minutes, not hours.
No charge.
In order to run this code you'll need python installed of course, plus download and install py-dns from http://pydns.sourceforge.net. Installing most python packages is dirt simple - usually involves no more than:
Code:python setup.py install
(used PHP hilighter just to make this more readable on-line; it is Python of course)
PHP Code:#!/usr/bin/env python
"""
$URL$
$Id$
A multi-threaded DNS query solution, originally developed to check a
large list of candidate domain names -- if no NS is present, the candidate
likely does not have a domain registration.
(C) mwatkins, a WebHostingTalk.com user. Use for any purpose you like provided
attribution is provided.
"""
# install PyDNS: http://pydns.sourceforge.net
# FreeBSD: /usr/ports/dns/py-dns
import DNS
import threading
import time
import Queue
from optparse import OptionParser
DNS.ParseResolvConf()
WORKERS = 10
def has_ns (domain):
"""(domain:string) -> boolean
"""
query = DNS.Request(domain, qtype="NS").req()
status = query.header['status']
return (domain, status)
def get_domains_from_file(path):
"""(path:string) -> [string]
Returns domains from a text file as a list of strings.
"""
return [domain.strip() for domain in open(path).readlines()
if domain.strip()]
class Worker(threading.Thread):
def __init__(self, work_queue, result_queue):
self.__queue = work_queue
self.__result = result_queue
threading.Thread.__init__(self)
def run(self):
while 1:
domain = self.__queue.get()
if domain is None:
break
self.__result.put(has_ns(domain))
def process_domains(domains):
"""(domains:sequence) -> ([string], [string])
Processes a sequence of domain strings and returns two lists
containing those that are potentially available for registration,
and those which have name servers defined and are therefore unavailable.
A final check for availability against a registrar whois API is recommended.
"""
work = Queue.Queue()
result = Queue.Queue()
for domain in domains:
work.put(domain)
# add work terminators
for i in range(WORKERS):
work.put(None)
workers = [Worker(work, result) for i in range(WORKERS)]
for worker in workers:
worker.start()
# wait for workers to complete
for w in workers:
w.join()
available = []
unavailable = []
while not result.empty():
domain, status = result.get(0)
if status == 'NXDOMAIN':
available.append(domain)
else:
unavailable.append(domain)
return available, unavailable
def run_domain_checker_main():
global WORKERS
parser = OptionParser()
parser.set_description('Checks domain names for available name servers. '
'Domain names with no name servers defined are potentially available '
'for registration. A final check for availability against a registrar '
'whois API is recommended.')
parser.add_option(
'-f', '--inputfile', dest='input_file', default='domains.txt',
help='A file containing one domain name per line ')
parser.add_option(
'-o', '--outputfile', dest='output_file', default='available.txt',
help=('A file to contain the candidate domains which may '
'be available. Note: This file is overwritten on every run.'))
parser.add_option(
'-s', '--server', dest='dns_server', default=None,
help='Force use of a specific name server, overriding /etc/resolve.conf.')
parser.add_option(
'-t', '--threads', dest='workers', default=10, type=int,
help='Number of worker threads')
parser.add_option(
'-c', '--cache', action="store_true", dest='use_cache', default=False,
help=('Use contents of <outputfile>, if it exists, as a cache. '
'In other words, do not requery domains that have been found '
'to be available. Newly found "available" domains will be '
'added to the cache in <outputfile>.'))
options, args = parser.parse_args()
domains = get_domains_from_file(options.input_file)
cached_domains = []
WORKERS = options.workers
if options.dns_server:
DNS.defaults['server'] = [options.dns_server,]
if options.use_cache:
try:
cached_domains = get_domains_from_file(options.output_file)
except IOError, e:
print "INFO: Skipping cache", e
# we don't want to check those that already are in the output file
domains = [domain for domain in domains if domain not in
cached_domains]
start = time.time()
print "Domains to process:", len(domains)
avail, unavail = process_domains(domains)
fout = open(options.output_file, 'w')
# we'll add cached_domains back to the output ([] if the option wasn't specified)
result = sorted(avail + cached_domains)
fout.writelines(['%s\n' % domain for domain in result])
fout.close()
print "Available domains written to file:", len(result)
print "Run time:", time.time() - start
if __name__ == '__main__':
run_domain_checker_main()
“Even those who arrange and design shrubberies are under
considerable economic stress at this period in history.”