href=
coupondirectory.net website that caused me so much grief showed that Google crawled many of its pages, but each and every one of them showed some robots.txt unavailable error. Reading on, I discovered that Google did crawl my
coupondirectory.net website, in fact
it crawled it many times, but before indexing any of my pages, it looked for the robots.txt file, but since I didnt have one, it couldnt access it. Also having received a 200-OK code instead of 404-not found, it assumed that robots.txt exists but is presently unavailable and therefore, rescheduled the crawl for a later time. In my case its been rescheduling it for a few months while I was desperately trying to promote my
coupondirectory.net website.
Imagine my reaction when I found out this. The evi.combination of always returning code 200-OK and absence of the robots.txt is deadly for search engine optimization. So if you are experiencing a similar situation, ask your web hosting provider whether they return any HTTP error codes, 404 in particular, and make sure you have the robots.txt file in your root directory.
"