Increasing DRT Speed?

[sashas]

Don't know if this is the right section.

I'm sure there are lots of DRT users here.

Any advice how I can increase its scanning speed? Takes an awful lot of time to do even 500 names.

I use proxies, the ones DRT provides everyday.

Would it be better to use paid proxies?

 

[Andrew Shaw]

The proxies should be just fine.

go under options/scan settings.

There you will see 5 tabs above. Click on the third one that says "threads"

1st - "Maximum simultaneous connections" This setting tells the system how many domain names will be scanned at one time. Setting this value to 5 will result in 5 domains being queried at a time. The higher this setting, the more bandwidth is required to process the query. Setting this value too high will result in * results and errors.

2nd Connection time-out refers to the number of seconds D.R.T. waits before considering the connection to be timed-out (remote server did not respond). The higher this setting the less retries will be required to receive results, as servers such as Overture tend to be slow.

3rd, Connection interval refers to the number of seconds D.R.T. Waits between making a new connection.

4th Maximum retries refers to the amount of retries allowed if a given connection failed. For example, if an Overture query timed out, D.R.T. Will retry it the specified amount of times.

Every system is different, and it will also depend on your internet connection. I have cable and can get away with 25, 2, 1/4, and 2. I could make it faster, but then the results arnt as complete.

Hope this helps.
Andrew-

 

[DomainMagnate]

If you only scan for availability scan without proxies and increase number of threads. I use 50, takes about an hour to scan 100K names on a dedicated server

 

[petrosc]

I just scanned 200,000 domains , it took about 8 hours.
You can increase the speed go to Options, Scan settings, whois/availability and disable all selected options. It took me a while to figure out that doing that would increase the speed at least 5x

oh yeah and in the scan settings - Search options select only "Availablility/Resolve only"

 

[Theo]

It's a Windows application, it can only do so much.
E.g. in 8 hours, I can scan for whois/availability almost 500k domains on a linux server.

 

[petrosc]

Not everybody needs Linux Theo the times you mention are pretty fast but 200k in 8 hours is not bad either

 

[actnow]

If you are scanning .org's or .info's, you can only go so fast.
.Org has a limit of 4 (or 5? or 6?) inquiries per permit GLOBALLY.

Since, Affilas runs the registry for .org, they have extended the .org policies towards the .info registry.
I just don't know what the new limitations are for .info. They took effect mid-2007.


add'l notation from .org.
"Effective 20 August 2005, WHOIS queries submitted through the Web-based WHOIS search mechanism
on the www.pir.org Web site are limited to 50 queries per minute." (This is globally.)

" PIR monitors all IP addresses accessing the .ORG WHOIS Port 43 server. All traffic will be logged,
and rate-limit validation logic will be applied to limit access by any given IP address
to a maximum of four queries per minute. If a unique IP address exceeds the limit,
the query will be stopped, and the error message will be displayed."

"When any user of the PIR Web site WHOIS service or the Port 43 WHOIS service has exceeded
the query limit for that service, ALL users will be blocked from using the PIR WHOIS service."

"PIR is committed to protecting the rights and privacy of registrants of .ORG domain names.
The previously unlimited access to Port 43 had led to abuses by speculators and data miners,
often using automated programs to make inquiries. Rate-limiting logic was introduced to reduce
this access and was based on consultation with ICANN and registrars."

 

[sashas]

thanks guys.

Will try these out.

Found some great names with good traffic yesterday. Couldn't be happier

 

[Theo]

Yes, PIR is "annoying" in that respect. But at least they implemented a restocking fee a long time ago, thus prohibiting kiters from benefiting indefinitely.

 

[jr360]

Petrosc,

I wouldn't mind if you'd touch base with me on D.R.T., as I'm not sure if i'm using it right.
My internet provider keeps restricting my us on internet for 24hrs??? after every update of my list in the system.
Any thoughts? or anyone else?
Not using paid proxies right??

Thanks in advance.
JR

 

[petrosc]

Paid proxies... yes of course mate I use paid proxies

myDRTproxy.com - $5/month

 

[Luc]

It's a Windows application, it can only do so much.
E.g. in 8 hours, I can scan for whois/availability almost 500k domains on a linux server.

I can do 10 million domains per day on a single redhat box with a 100mbit pipe. That of course is registry level queries only.

As far as PIR, DRT supports performing proxy based ORG/DE/INFO, etc queries, meaning that it can use web proxies to query the registry, making it easier to circumvent the silly 4 domains/minute restriction.

Luc

 

[jdk]

How do you do queries on a linux server? I would be interested in purchasing such a script.

 

[tonyk2000]

If you are using windows xp SP2 or vista, the scan speed may also be limited by windows operating system. It would be the case in case of connection attempts to dead or slow proxies. I was able to increase the speed using the following patch. It is for windows xp. Vista patch may or may not exist elsewhere. NOTE: You may need to re-apply the patch after each security update downloaded from microsoft (if tcpip.sys was changed).



http://www.lvllord.de/?lang=en&url=4226patch/faq

Quote:

What's this EventID 4226?
The EventID 4226 means, that there are to many concurrent TCP connection attempts. A connection attemp is a query to a computer, if it accept the TCP connection or not. If the computer is for example no more existent, it will be waited for a few seconds, until a timeout occurs and the connection attemp get canceled.
Since XP SP2 there are only 10 concurrent TCP connection attempts possible, while in SP1 it has not been limited.

Which effects does this limit have?
Applications with many connection attemps may work bad or not at all. Even web browsers, eMail clients or antivir programs can be affected and might bring a long time of waiting.
Also administrative diagnosticprograms in companies make problem. In short, there seems to be more problems than there is an advantage.

How can I find out, if I'm affected?
Just have a look at Start/Maintenance/Control Panel/Administration Management/Eventview/System if there are entries with the event id 4226. If yes, minimum one time the limit has been reached

I read something, that it's possible to change limit via registry (TcpNumConnections). Is that true?
Unfortunately not. Because the concurrent connection attemp limit has nothing to with concurrent connections, this registry-key is useless. Unfortunately there is no registry-key, which would allow the user to change the concurrent connection attemps.

50 concurrent, half-open connections is to less for me. Is it possible to get more?
Yes! Just use as a Parameter /L=limit, where limit describes the new limit.
For example a 100 limit can be set with the command:
EvID4226Patch /L=100

Which other parameters does exist?
Not entered yet ...

Which languages are supported?
Theoretical every lange is supported. If not, please contact me!

I have Windows 95/98/ME/2000/XP SP1. Will these get supported soon, too?
Because the limit got introducted in XP SP2, the other operating systems are currently unlimited.
Exception: Windows 2003 Server seems to be limited as well since SP1 beta. An update will follow ...

 

[domain54326]

How do I turn off the apprasial feature?

 

[italiandragon]

since when DRT offers Proxies too ???

Thanks

I think we should make a thread where discuss all tips and tricks to use it better

Thanks again

 

[jdk]

Under Proxy Settings then Proxy Server Sources

The question is how often are they updated with new proxies?

 

[italiandragon]

Under Proxy Settings then Proxy Server Sources

The question is how often are they updated with new proxies?

Thanks!!!

 

[tristanperry]

I think we should make a thread where discuss all tips and tricks to use it better

Thanks again

*Nods* I'd like that, as a DRT newbie (as of yesterday)

My main question/concern is thus. I've followed petrosc's advice by the word, and at first my scans were scanning at great speeds (I estimated it'd end up at around 400,000 domains would be scanned in 8 hours).

However after a while things slowed. A lot. Things went down to about 0.5 - 2 domains per second, which isn't great considering I was trying to scan 456k domains.

I tried out DRT's proxy list, and I scanned the list, and ended up with 48 anonymous proxies.

Any ideas why things went so slow? And how can I speed things up?

EDIT: I just tried out MyDRTProxy.com, although... what's the point in it? Nearly all the proxies in today's list are dead?

 

[ATP]

Why domain portfolio isn't automatically loading on start?