quick question about robot.txt files

[Guest]

hey guys and gals

 

i am trying to get some high results on Google, yahoo and all the other ones. I am using all the contributions, looking on the net, looking for good keywords and titles, adding my site to the crawlers and so on. i was going to download and install Dynamic SiteMap V 1.0 but they want me to create a robot.txt. I am not very familiar with the file but i do know that it helps the crawler navigate your site faster and easier (i think). i also heard that if you don’t know what you are doing with this file it could kill your site completely. If some one could enlighten me about this file while i still such on the web that would be great.

 

also i am doing this because i am searchable (not very high thou) on google but not yahoo. i submitted my site and see that it is verified but it also says that the numbers if pages indexed is 0. Now i know that is bad. i just dont know how to change it. i was going t create a sitemap and submit it to yahoo and hopefully that with change the index. So help please, anything would be great

[germ]

A lot of info here.

 

Just pull up your favorite search engine and search for robots.txt

[Guest]

A lot of info here.

 

Just pull up your favorite search engine and search for robots.txt

 

 

 

are robot.txt files a good idea and should be used or are they like meta tags keywords? They are a good idea but not very usefull

[germ]

An example for a "root" installed store:

 

User-agent: *
Disallow: /account.php
Disallow: /account_edit.php
Disallow: /account_history.php
Disallow: /account_history_info.php
Disallow: /account_password.php
Disallow: /add_checkout_success.php
Disallow: /address_book.php
Disallow: /address_book_process.php
Disallow: /advanced_search.php
Disallow: /checkout_confirmation.php
Disallow: /checkout_payment.php
Disallow: /checkout_payment_address.php
Disallow: /checkout_process.php
Disallow: /checkout_shipping.php
Disallow: /checkout_shipping_address.php
Disallow: /checkout_success.php
Disallow: /contact_bean.php
Disallow: /cookie_usage.php
Disallow: /create_account.php
Disallow: /create_account_success.php
Disallow: /login.php
Disallow: /password_forgotten.php
Disallow: /popup_image.php
Disallow: /shopping_cart.php
Disallow: /product_reviews_write.php

It's just used to keep "good robots" from entering/indexing pages you don't want them to.

 

Keep in mind that this behavior is strictly voluntary.

 

Normal web robots only follow links in pages.

 

If you don't have a link to something in your store (like the admin) don't place it in your robots.txt file

 

"Bad robots" (ones that don't follow the rules in robots.txt) hit the places you disallow first.

 

Don't tip your hand to them and tell them about places they couldn't find by normal means.

[Guest]

The easiest way is to sign up for Google Webmaster Tools and use the robots.txt generator.

 

Tip! The robots.txt is used to restrict the crawlers from crawling files and folders on your site. It's different to the xml site map that does the opposite.

[Guest]

An example for a "root" installed store:

 

User-agent: *
Disallow: /account.php
Disallow: /account_edit.php
Disallow: /account_history.php
Disallow: /account_history_info.php
Disallow: /account_password.php
Disallow: /add_checkout_success.php
Disallow: /address_book.php
Disallow: /address_book_process.php
Disallow: /advanced_search.php
Disallow: /checkout_confirmation.php
Disallow: /checkout_payment.php
Disallow: /checkout_payment_address.php
Disallow: /checkout_process.php
Disallow: /checkout_shipping.php
Disallow: /checkout_shipping_address.php
Disallow: /checkout_success.php
Disallow: /contact_bean.php
Disallow: /cookie_usage.php
Disallow: /create_account.php
Disallow: /create_account_success.php
Disallow: /login.php
Disallow: /password_forgotten.php
Disallow: /popup_image.php
Disallow: /shopping_cart.php
Disallow: /product_reviews_write.php

It's just used to keep "good robots" from entering/indexing pages you don't want them to.

 

Keep in mind that this behavior is strictly voluntary.

 

Normal web robots only follow links in pages.

 

If you don't have a link to something in your store (like the admin) don't place it in your robots.txt file

 

"Bad robots" (ones that don't follow the rules in robots.txt) hit the places you disallow first.

 

Don't tip your hand to them and tell them about places they couldn't find by normal means.

Where did contact_bean.php come from? I don't see that file anywhere in my install. Am I missing something?

[germ]

Congrats!!!

 

You're the first one to notice.

 

You win a cookie!

:)

 

Actually that's a screwup.

:blush:

 

Should be:

 

Disallow: /contact_us.php