robot.txt file: Is it needed?

[Trentide]

I'm trying to find info regarding this but cant seem to find any in the forums.

 

I have an updated spiders.txt file. Do these two files fulfill the same purpose?

 

I also have the google sitemap contribution to help Google catalog my site.

 

Do I really need a robot.txt file?

 

Thanks In Advance,

Jason

[bill110]

The robots.text file tells the bots which files or directories NOT to go to. No sense in indexing login, shopping cart, etc.

[spax]

robots.txt denies spider access to whatever directory/file you specify. So long as it is a nice spider.

 

spiders.txt is osC specific. If Prevent Spider Sessions is enabled in your admin, osC wont start a session for a spider identified by the user agent string, if listed in the spiders.txt file.

 

Read this:

 

http://www.oscommerce.com/forums/index.php?s=&...st&p=549863

[Trentide]

Thanks for responding. I appreciate your input!

 

Where do I get a robot.txt file and how do I go about configuring it?

 

Jason

[Jack_mcs]

It is just a .txt file that goes in the roor of your shop. There are examples here on the forum (search for robots), a contibution with an example in it or you can go to any site that has one and type in http://somesite.com/robots.txt to get a copy. Just change the entries to match your site setup.

 

And just to clarify, the robots file only asks the search bots to not enter those areas. If they choose to enter anyway, and some will, they will ignore the robots file. But for the ones that follow the rules, like the major search engines, it is something you still need.

 

Jack

[Bushmaster]

Thanks for responding. I appreciate your input!

 

Where do I get a robot.txt file and how do I go about configuring it?

 

Jason

 

I do not know if this contains all the spots you do not want indexed or may have to much in it. Just make a file called robots.txt with word pad or other text editor and copy and past this in it, then save to your catalog directory or the root of your store.

User-agent: Googlebot-Image

Disallow: /

 

User-agent: *

Disallow: /account.php

Disallow: /advanced_search.php

Disallow: /checkout_shipping.php

Disallow: /cookie_usage.php

Disallow: /create_account.php

Disallow: /login.php

Disallow: /password_forgotten.php

Disallow: /popup_image.php

Disallow: /shopping_cart.php

Disallow: /images/

Disallow: /cgi-bin/

Disallow: /download/

Disallow: /product_reviews_write.php

Disallow: /validation_png.php

Disallow: /validation_png_2.5.php

Disallow: /popup_cvv.php

Disallow: /cvv_help.php

Disallow: /address_book.php

Disallow: /account_password.php