Jump to content
  • Checkout
  • Login
  • Get in touch

osCommerce

The e-commerce.

Googlebot ignoring Robots.txt ?!


chrisab

Recommended Posts

Hi,

I've got robots file:

Disallow: /includes

Disallow: /cgi-bin

Disallow: /account.php

Disallow: /account_edit.php

Disallow: /account_history.php

Disallow: /account_history_info.php

Disallow: /account_password.php

Disallow: /add_checkout_success.php

Disallow: /address_book.php

Disallow: /address_book_process.php

Disallow: /advanced_search.php

Disallow: /checkout_confirmation.php

Disallow: /checkout_payment.php

Disallow: /checkout_payment_address.php

Disallow: /checkout_process.php

Disallow: /checkout_shipping.php

Disallow: /checkout_shipping_address.php

Disallow: /checkout_success.php

Disallow: /cookie_usage.php

Disallow: /create_account.php

Disallow: /create_account_success.php

Disallow: /login.php

Disallow: /password_forgotten.php

Disallow: /popup_image.php

Disallow: /shopping_cart.php

Disallow: /product_reviews_write.php

 

If I go to google webmaster and 'fetch as googlebot' say .co.uk\shopping_cart.php it fetches the whole page, any reason why robots is not working?!

 

Thanks

Chris

Link to comment
Share on other sites

If that is the whole file, you are missing the user agent line. Google has a robots checker that should show the problem.

Support Links:

For Hire: Contact me for anything you need help with for your shop: upgrading, hosting, repairs, code written, etc.

All of My Addons

Get the latest versions of my addons

Recommended SEO Addons

Link to comment
Share on other sites

Ahh good point! I downloaded the file from the robots contribution on here!

 

I've changed to this, google still reading it but presume it needs time to re-read robots.txt?

 

User-agent: *

Disallow: /includes

Disallow: /cgi-bin

Disallow: /account.php

Disallow: /account_edit.php

Disallow: /account_history.php

Disallow: /account_history_info.php

Disallow: /account_password.php

Disallow: /add_checkout_success.php

Disallow: /address_book.php

Disallow: /address_book_process.php

Disallow: /advanced_search.php

Disallow: /checkout_confirmation.php

Disallow: /checkout_payment.php

Disallow: /checkout_payment_address.php

Disallow: /checkout_process.php

Disallow: /checkout_shipping.php

Disallow: /checkout_shipping_address.php

Disallow: /checkout_success.php

Disallow: /cookie_usage.php

Disallow: /create_account.php

Disallow: /create_account_success.php

Disallow: /login.php

Disallow: /password_forgotten.php

Disallow: /popup_image.php

Disallow: /shopping_cart.php

Disallow: /product_reviews_write.php

Link to comment
Share on other sites

  • 6 months later...

Ahh good point! I downloaded the file from the robots contribution on here!

 

I've changed to this, google still reading it but presume it needs time to re-read robots.txt?

 

User-agent: *

Disallow: /includes

Disallow: /cgi-bin

Disallow: /account.php

Disallow: /account_edit.php

Disallow: /account_history.php

Disallow: /account_history_info.php

Disallow: /account_password.php

Disallow: /add_checkout_success.php

Disallow: /address_book.php

Disallow: /address_book_process.php

Disallow: /advanced_search.php

Disallow: /checkout_confirmation.php

Disallow: /checkout_payment.php

Disallow: /checkout_payment_address.php

Disallow: /checkout_process.php

Disallow: /checkout_shipping.php

Disallow: /checkout_shipping_address.php

Disallow: /checkout_success.php

Disallow: /cookie_usage.php

Disallow: /create_account.php

Disallow: /create_account_success.php

Disallow: /login.php

Disallow: /password_forgotten.php

Disallow: /popup_image.php

Disallow: /shopping_cart.php

Disallow: /product_reviews_write.php

 

I think you need a / after:

 

change:

Disallow: /includes

Disallow: /cgi-bin

 

to

 

Disallow: /includes/

Disallow: /cgi-bin/

 

according to Google webmaster support:

  • To block a directory and everything in it, follow the directory name with a forward slash.
    Disallow: /junk-directory/
  • To block a page, list the page.
    Disallow: /private_file.html

 

I don't know that I would include

Disallow: /admin/

 

which should be renamed to something obscure anyway

 

suppose you can disallow the new obscure name, but you may just be pointing bad bots to the place you don't want them

 

DO ADD your sitemap(s) to the robots.txt

 

Sitemap: http:// www. yoursite. com/sitemap.xml

Sitemap: http:// www. yoursite. com/sitemap2.xml

Web Developer, Firebug, and Notepad++ are powerful free tools for web design.

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...