Sincraft Posted February 12, 2005 Share Posted February 12, 2005 I am going to post my robots file in hopes someone here with good SEO experience and osCommerce can tell me if this looks correct or not. I keep logging into my who's online page and keep finding MSNbot crawling around. It basically lives there day and night with a break in the middle of the day, which I guess is a good thing. But I often find it in , account.php or shopping_cart.php etc...so I disallowed a few things, but tonight say that it was in COOKIE_USAGE.PHP... How does it get into that without a link to it? I can't even figure out how it got there. So anyway, I wanted to know the above question for curiousity reasons and wanted someone to comment on this robot.txt.. THis is my ENTIRE robot.txt contents. User-agent: * Disallow: /cgi-bin/ Disallow: /usage/ Disallow: /images/ Disallow: /(censoredadmincat)/ Disallow: /tmp/ Disallow: /account.php Disallow: /shopping_cart.php Disallow: /checkout_shipping.php Disallow: /login.php Disallow: /create_account.php Disallow: /advanced_search.php Disallow: /product_review.php Disallow: /popup_image.php Disallow: /cookie_usage.php what do you think? Should I also consider putting an * after the .php just incase there are session ids etc as many of my urls are not wrapped with Ultimate SEO URLs (like privacy.php and articles.php&page2_how to buy etc. I noticed it is still looking at a popup image as we speak even though I have it commented out in my robots file. /popup_image.php?pID=43 (for example) S Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.