We need to monitor hundreds of thousands of domains per month - Looking for paid help

[DomainEmpire.com]

I see there're companies able to monitor the status of all domains in zone files (at least for com/net/org) and they're even able to save every few days the whole whois database, how are they doing that ?

Well, we need a solution for a smaller problem: we need a server side software/script or any other way to monitor the registration status of some hundreds of thousands of domains.
Basically, a fulltime automated monitoring solution, to get a daily list of all entered domains which changed their registration status in the latest 24hours.

We're obviously looking for a "paid help"; for more details and a more clear idea of our needs and expectations, email us at: domainsales (at) domainshowcase.com if you think to have the right solution in mind, thank you.

 

[amplify]

The company probably has access to the .com/.net zone file

http://www.verisigninc.com/en_US/pr...registry-products/tld-zone-access/index.xhtml

And the .org zone file

http://www.pir.org/help/access

Then, they go through it and take the info they want and use it to their benefit (drop catching or whatever).\

And you're not going to be able to do 100,000+ whois to get that information as GoDaddy blocks it with a captcha as do other registrars.

 

[DomainEmpire.com]

I've some concerns at this point, just curious to understand more:

1. Companies like domaintools.com offers an archive of past whois for all com/net/org domains, for each domain they save new records every few weeks, we're members there, we queries multiple domains and yes, they've multiple records for each of them, no exception. How can they do this ? Does this mean they've a way to check/save whois records for all names available in zone files ?
Yes, that's correct, they save complete whois records (including registrant details, exp dates, domain status and dns) ...

2. In the near past there was a spammer with the technology to check daily whois for all newly registered domains and to email to any registrant (independently by the registered domain) his link to invite them to check their website, to find better available domains related to the topic of the registered one so this's another example of "someone" able to get whois records of dozens of thousands of domains per day.

Briefly, if in a far future we should become a registrar, should we have the necessary tools to perform this kind of activity ?
I'm just trying to learn more about this topic, thanks for helping to clarify my doubts.

 

[aclass]

1. It might be done by using many many IPs so you don't get your IP blocked for too many whois requests. The programming part is easy. Just the IP part for such a big number of domains might get a bit pricey.

2. There are still such spammers as I still get an email a couple of days after I register a domain.

 

[amplify]

IPs won't help when you get a GD domain, though won't flag you for spam, if you're using a human verified captcha service (a couple bucks per 1000).

 

[katherine]

1. Companies like domaintools.com offers an archive of past whois for all com/net/org domains, for each domain they save new records every few weeks, we're members there, we queries multiple domains and yes, they've multiple records for each of them, no exception. How can they do this ? Does this mean they've a way to check/save whois records for all names available in zone files ?

In the zone files, you will only get the list of domain names with associated name servers. Nothing more.
So just by comparing zone files on a daily basis for example, you can see which domain names have changed name servers. You can also see names added/removed (typically new registrations/dropping domains).

For .com/.net the full whois record (with registrant information) is maintained at the registrars.
For .org the information is available at registry level.

Scraping whois data en masse is difficult because access is capped so your IP address will be quickly banned after more than n queries per minutes. To get around this you typically need a lot of proxies.
How domaintools.com does that ?
Well the registrars often have mutual arrangements to whitelist designated IP addresses, and domaintools.com is a registrar.

 

[DomainEmpire.com]

Thanks for these detailed info, I'm continuing to learn.

Well, basically what we really need is much simple to do I think.
We need a way to monitor on a daily basis "domain status & dns" of a list of about 400'000 domains (mainly com/net, around 20% of .org).

Briefly, we'd use a remote server and we'd pay for the necessary resources, we'd need a "stable software" which should automatically perform this daily task and save in a txt file the list of all domains which added any of the "given keywords" (entered by us, of course) in the just changed dns or status.

For example, all domains where the keyword "renew" has been added in their dns or the keyword "hold" in their status.
We're collecting few quotes, hoping to get some professional help in realizing this project.

Feel free to PM or email if you think to be able to help, thanks.