stiksandstones Posted August 5, 2005 Share Posted August 5, 2005 I have my robots.txt file in main root of my site. I was checking my logs and find it says 66.249.64.52 - - [01/Aug/2005:23:17:00 -0400] "GET /robots.txt HTTP/1.0" 200 632 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)" 66.249.64.52 - - [01/Aug/2005:23:17:00 -0400] "GET / HTTP/1.0" 200 6397 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)" I know the 200 code means it was a success but nothing more, isnt it supposed to crawl ALL my pages? looks like it didnt crawl any. So then I checked my logs for my subdomain (store.mysite.com) and found... 66.249.64.16 - - [28/Mar/2005:21:38:50 -0800] "GET /robots.txt HTTP/1.0" 404 - "-" "Googlebot/2.1 (+http://www.google.com/bot.html)" 66.249.64.16 - - [28/Mar/2005:21:38:57 -0800] "GET / HTTP/1.0" 200 14980 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)" 404 means error, didnt find the robots.txt in my http://store.mysite.com Am I supposed to have a robots in my subdomain AND my main domain? Link to comment Share on other sites More sharing options...
Guest Posted August 5, 2005 Share Posted August 5, 2005 a subdomain is different than a domain Link to comment Share on other sites More sharing options...
stiksandstones Posted August 5, 2005 Author Share Posted August 5, 2005 a subdomain is different than a domain <{POST_SNAPBACK}> Very informative! So does anyone know if I should have a robots.txt in my main site (www.mysite.com) AND in my subdomain (http://store.mysite.com)? The robots.txt has disallows for /store/admin etc.... but I just would like to know if I should have one on the subdomain. AND am I supposed to see the bots/spiders crawling all my dynamic php pages? Thanks Link to comment Share on other sites More sharing options...
FalseDawn Posted August 5, 2005 Share Posted August 5, 2005 It depends how you have your subdomain configured - if it is truly a separate domain with its own DNS entry, then yes, if it accessed through your main domain via redirects, then no. Bots will crawl dynamic pages, yes - but they won't crawl them all at once - it takes several weeks to get fully indexed (at least). Link to comment Share on other sites More sharing options...
stiksandstones Posted August 5, 2005 Author Share Posted August 5, 2005 It depends how you have your subdomain configured - if it is truly a separate domain with its own DNS entry, then yes, if it accessed through your main domain via redirects, then no.Bots will crawl dynamic pages, yes - but they won't crawl them all at once - it takes several weeks to get fully indexed (at least). <{POST_SNAPBACK}> domain and subdomain are on same IP...as for DNS, I have only entered in my main domain info. But my google ads point to store.mysite.com Is it strange the bots are making seperate trips to store/ and root? Link to comment Share on other sites More sharing options...
FalseDawn Posted August 5, 2005 Share Posted August 5, 2005 Not really - it depends on what inbound links you have. In your configuration, you will only need 1 robots.txt file in your public_html directory. When a bot accesses your subdomain, then (if it is behaving itself), it should retrieve the robots.txt file from your root folder to see where it is allowed to go. Link to comment Share on other sites More sharing options...
New 
Recommended Posts
Archived
This topic is now archived and is closed to further replies.