Googlebot and other spiders -- Robots.txt question

[dchism]

Ok, I just recently created a robots.txt file for the first time. I'm still a noob at this. I have two issues:

 

1.) Where must the robots.txt reside? In the root of the www directory?

 

2.) I've noticed that googlebot (the only one I've noticed doing this so far) is creating a cart even though shopping cart.php is disallowed. Not only is it creating a cart, but it is putting thousands of $$ worth of items in it. Because of this, it takes over 5 minutes to bring up my "who's online" (with the enhancement contrib). I'm sure this is what is causing the slowness because when the bot isn't there with a packed cart, there are no delays in who's online. Last night, I killed the bot session but it came right back and packed the cart full again. It is still there now and I can't get rid of it. Any suggestions would be great. Here is my robots.txt. I have the mini stores contrib installed so there are multiple "catalogs"...

 

# osCommerce robots.txt

# Currently disallow all shop stuff to the Google Image bot
# Mainly image hunters anyway, they eat up bandwidth...
User-agent: Googlebot-Image
Disallow: /cgi-bin/
Disallow: /catalog/
Disallow: /homeschool/
Disallow: /classroom/
Disallow: /images/
Disallow: /images2/
Disallow: /classstore/

# ALL search engine spiders/crawlers (put at end of file)
User-agent: *
Disallow: /cgi-bin/
Disallow: /catalog/
Disallow: /homeschool/download/
Disallow: /homeschool/admin/
Disallow: /homeschool/pub/
Disallow: /homeschool/account.php
Disallow: /homeschool/advanced_search.php
Disallow: /homeschool/checkout_shipping.php
Disallow: /homeschool/create_account.php
Disallow: /homeschool/login.php
Disallow: /homeschool/password_forgotten.php
Disallow: /homeschool/popup_image.php
Disallow: /homeschool/shopping_cart.php
Disallow: /classroom/download/
Disallow: /classroom/admin/
Disallow: /classroom/pub/
Disallow: /classroom/account.php
Disallow: /classroom/advanced_search.php
Disallow: /classroom/checkout_shipping.php
Disallow: /classroom/create_account.php
Disallow: /classroom/login.php
Disallow: /classroom/password_forgotten.php
Disallow: /classroom/popup_image.php
Disallow: /classroom/shopping_cart.php
Disallow: /classstore/admin/
Disallow: /classstore/download/
Disallow: /classstore/pub/
Disallow: /classstore/account.php
Disallow: /classstore/advanced_search.php
Disallow: /classstore/checkout_shipping.php
Disallow: /classstore/create_account.php
Disallow: /classstore/login.php
Disallow: /classstore/password_forgotten.php
Disallow: /classstore/popup_image.php
Disallow: /classstore/shopping_cart.php
Disallow: /orchardold/
Disallow: /orchardnew/
Disallow: /adminlinks/
Disallow: /bjuimage/
Disallow: /classes/
Disallow: /cornerstone/
Disallow: /news/
Disallow: /newsccstore/
Disallow: /newsproductstore/
Disallow: /stats/
Disallow: /productstore/
Disallow: /productstore1/
Disallow: /teacherstore/
Disallow: /test/
Disallow: /vcalendar/

[abra123cadabra]

You should change your settings in your admin panel to prevent spider session ids. That should take care of any spider filling a cart. There is a contribution that is regularly updated with new spiders and bots.

 

abra

[dchism]

Great! Thank you!