Best Domain Availability Checking Apis?

[Kadence]

I have long keyword lists that I'd like to check. Up to this point up been using the Godaddy bulk tool but as I check .com/.net/.org domains with long lists, the task gets tedious.

I'd like to make myself a program that checks automatically. I think some people have software like this, but they use the Godaddy bulk checker and basically scrape the results.

An API would be my preference. There's the WhoisAPI - but for bulk checking that is enormously expensive.

After googling I found "Free Domain API" and that looks good at 50 requests/minute. But I was wondering about any other alternatives, as it seems they will not be free forever, plus they claim "98% accuracy" which seems odd to me - why would the accuracy be so relatively poor?

Namecheap and Name have APIs but I don't know what their restrictions might be.

If anyone has any experience with these APIs please share your thoughts about them

And if there's any other good APIs I'd be interesting in hearing about them too. Paid is fine, as long as it's substantially cheaper than the WhoisAPI!

 

[katherine]

I have a trick to speed up the work It's a bit technical but it works like this:
rather than check domain availability the 'classic' way, you could have a script that does a DNS query against each domain.
For example: in Linux the command line

dig -t ns domain.com

returns something like this:

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1 <<>> -t ns domain.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50682
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;domain.com.                    IN      NS

;; ANSWER SECTION:
domain.com.             60      IN      NS      ns3.domain.com.
domain.com.             60      IN      NS      ns2.domain.com.
...

It's the list of name servers defined for the domain name.

Short version:

dig -t ns +short domain.com

Result:

ns3.domain.com.
ns2.domain.com.

NS: the output may be polluted by error messages, you have to handle errors.

Two possible outcomes:

1. at least one name server returned => the domain exists
2. no name servers returned => the domain does not exist OR the domain has no name servers OR the domain name is suspended, expiring OR DNS error...

You don't have to do the command line, you can use other scripts or libraries instead.
DNS queries are fast, and there are no quotas unlike whois lookups. They can (should) be performed in parallel, which will dramatically the time needed to complete the task...

Then you only have to check the domains for which no NS were returned, to double check their status. That list will be significantly shorter. You get the idea.

 

[Kadence]

Hey thanks for that, that seems like a good idea

When doing 'dig' for some non-existent domain names, for the IN SOA section I'm getting "a.gtld-servers.net. nstld.verisign-grs.com." Tried thris from a couple different linux machines and got the same. Is this something that will be the same for everyone based on TLD being checked?

Or I suppose in other words, for a failed check for a given TLD, will the dig response always contain the same message?

 

[katherine]

For non-existent domains you'll get an NXDOMAIN response like this:

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1 <<>> -t ns p.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 26644
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

But dig will not return an error in this case.

So I think you should look at both response and error code.
Error codes are as follows:

0: Everything went well, including things like NXDOMAIN
1: Usage error
8: Couldn't open batch file
9: No reply from server
10: Internal error

Source: http://linux.die.net/man/1/dig

So if you do dig -t ns +short <some non-existent domain>, you'll a return code of 0 (=OK) but no NS.
For scripting I would stick to dig -t ns +short, because it's easier to interpret the results programmatically.

To sum up there are 3 possible outcomes:

1. return code !=0 => an error occured
2. return code =0, NS returned => domain is taken
3. return code =0, no NS returned => domain does not exist, or has no name server, or is in redemption etc.

You only have to double-check domains that fall under 1. or 3.

If you do something like:

dig -t ns +short .com; echo $?

You'll get:

dig: '.com' is not a legal name (unexpected end of input)
10

You can suppress the error messages like this:

dig -t ns +short .com > /dev/null 2>&1; echo $?

Result:

10

A quick script would be along these lines:

#!/bin/bash

export LC_ALL=C

echo -n "Domain to check: "
read domain
echo "Domain: $domain"

ns=$(dig -t ns +short "$domain" 2> /dev/null)
err=$?
if [ "$err" -ne 0 ]; then
        echo "An error occured (code: $err)"
else
        # response empty ?
        if [ ! -z "$ns" ]; then
                echo -n "NS found: "
                echo "$ns" | tr "\n" " "
        else
                echo "No NS found"
        fi
fi

It should handle the three possible situations and you could output the results to a CSV file.
Again, this is quick & dirty code for demonstration purposes.

 

[Kadence]

Oh wow. Thanks for the really informative response! I'll try that.

 

[katherine]

Here is a reviewed version of the script.

Say you have a list of domains to check in a file domains.txt like this:

thisdomainsdoesnot-exist.com
.com
test.com
cctld.ru

NB: domains can be in any TLD.

This code will process the input file and create a results file domains.txt.results.csv in the same directory.

#!/bin/bash

export LC_ALL=C
echo -n "List to check: "
read inputfile
echo "List: $inputfile"

if [ ! -f "$inputfile" ]; then
        echo "File $inputfile does not exist, exit"
        exit 0
else
        filename=$(basename "$inputfile")
        outputfile=$(dirname "$inputfile")/"$filename.results.csv"
        echo "Output file: $outputfile"
fi

echo "domain,error,status,NS"  >  "$outputfile"
while read domain
do
        echo "Processing domain: $domain"
        ns=$(dig -t ns +short   "$domain" 2> /dev/null)
        err=$?
        if [ "$err" -ne 0 ]; then
                echo "An error occured (code: $err)"
                echo "$domain,$err,ERR"  >>  "$outputfile"
        else
                # response empty ?
                if [ ! -z "$ns" ]; then
                        echo -n "NS found: "
                        nslist=$(echo "$ns" | tr "\n" " ")
                        echo "$nslist"
                        echo "$domain,$err,NS,$nslist"  >>  "$outputfile"
                else
                        echo "No NS found"
                        echo "$domain,$err,NXD"  >>  "$outputfile"
                fi
        fi

done < $inputfile

Result:

domain,error,status,NS
thisdomainsdoesnot-exist.com,0,NXD
.com,10,ERR
test.com,0,NS,ns66.worldnic.com. ns65.worldnic.com.
cctld.ru,0,NS,ns.cctld.ru. ns1.cctld.ru.

Now you have a neat file in CSV format.

• First column: the domain name
• Second column: error code returned by dig (0 = OK)
• Third column: status:
  1. ERR: error occured
  2. NXD: NXDOMAIN
  3. NS: name servers found, domain exists
• Fourth column: the list of name servers if any

To make the process more efficient, multiple DNS checks should be done in parallel by spawning the process into multiple threads (using xargs for example).
Your homework