Does OSC include robots.txt?

[budbiss]

Am I supposed to create and upload robots.txt myself?

[Guest]

Yes, its up to you to create one. If you need help making one, do a search, there is plenty out there to help. Linda's SEC for MS2 had a good one in it if you can't find one, but it would need to be customized for your site.

 

HTH

[budbiss]

This might be a stupid question, but what would I not want to allow to come to my site? I am pretty sure I want everything to be allowed to access everything.

[Guest]

You don't want spiders going to certain folders/files on your site. This is a good way to prevent that.

 

For example, here is a piece of my robots.txt which I got from Linda's SEC

 

#Block out things that are secure or login oriented
Disallow: /account.php
Disallow: /account_edit.php
Disallow: /account_history.php
Disallow: /account_history_info.php
Disallow: /account_newsletters.php
Disallow: /account_notifications.php
Disallow: /account_password.php
Disallow: /address_book.php
Disallow: /address_book_process.php
Disallow: /advanced_search.php
Disallow: /advanced_search_result.php
Disallow: /check_secure.php
Disallow: /checkout_confirmation.php
Disallow: /checkout_payment.php
Disallow: /checkout_payment_address.php
Disallow: /checkout_process.php
Disallow: /checkout_shipping.php
Disallow: /checkout_shipping_address.php
Disallow: /checkout_success.php
Disallow: /create_account.php
Disallow: /create_account_success.php
Disallow: /download.php
Disallow: /info_shopping_cart.php
Disallow: /login.php
Disallow: /logoff.php
Disallow: /password_forgotten.php
Disallow: /popup_image.php
Disallow: /popup_search_help.php
Disallow: /product_reviews_write.php
Disallow: /redirect.php
Disallow: /shopping_cart.php
Disallow: /tell_a_friend.php
#

 

This is files only, you should do this for folders and other items you don't want indexed by spiders.

[budbiss]

ok, i see now...i don't want google getting access to customer info or things like that so i would want to put some disallows in there

 

BTW, can anyone recommend a good tutorial for writing robots?