- Joined
- Nov 24, 2007
- Messages
- 768
- Reaction score
- 1
I was looking for the history of a .COM that I own. (I bought it recently and put up a blog. I went to http://www.archive.org trying to make sure it doesn't have a negative history for me to clean up.)
Their site tells me it cannot find any info on my site due to a Robots.txt file. It says "We're sorry, access to basicpolitics.com has been blocked by the site owner via robots.txt. Their FAQ page says the robots.txt file forbids their web crawler from archiving any domain with such a file.
It offers me a link to view the Robots.txt file, but when I click on it, I get this, "Failed Connection. We're sorry. Your request failed to connect to our servers ."
I have searched all of the files on my server and do not find such a file. I never placed one there and I don't think anyone else has, either.
A previous owner may have had a Robots.txt file that is causing them to continue by-passing my site when their spiders are wandering about.
I sent them an e-mail to [email protected] asking them how to get my site off their list of sites they don't archive. It has been several days and I have had no response.
Has anyone else encountered this?
How do I get my site off the list of what they don't archive?
I am the registrant of the site now, and I wish to get it archived, and otherwise taken notice of so I can drive more traffic to it.
Thank you in advance for your help.
toria
Their site tells me it cannot find any info on my site due to a Robots.txt file. It says "We're sorry, access to basicpolitics.com has been blocked by the site owner via robots.txt. Their FAQ page says the robots.txt file forbids their web crawler from archiving any domain with such a file.
It offers me a link to view the Robots.txt file, but when I click on it, I get this, "Failed Connection. We're sorry. Your request failed to connect to our servers ."
I have searched all of the files on my server and do not find such a file. I never placed one there and I don't think anyone else has, either.
A previous owner may have had a Robots.txt file that is causing them to continue by-passing my site when their spiders are wandering about.
I sent them an e-mail to [email protected] asking them how to get my site off their list of sites they don't archive. It has been several days and I have had no response.
Has anyone else encountered this?
How do I get my site off the list of what they don't archive?
I am the registrant of the site now, and I wish to get it archived, and otherwise taken notice of so I can drive more traffic to it.
Thank you in advance for your help.
toria