Jump to content
  • Checkout
  • Login
  • Get in touch

osCommerce

The e-commerce.

can someone tell me how to do the robots.txt?


hpqnet

Recommended Posts

Hi, I have been slammed since day one on my bandwidth. I am using about 3GB a month, and I have got to do something about it. I did have to upgrade to 2GB a month for an extra $5 a month, but this also doubled my disk storage, so it was not so bad. I do not have a robots.txt file on my site and I am assuming this would help cut down on some of the bandwidth usage, right?

 

Can someone help?

 

I am using osc2.2ms2 and I have my catalog setup under / instead of the /catalog

my live site

Link to comment
Share on other sites

Take a look here -> Robots. It should answer all of your questions and provides an example of a robots.txt file too. You'll want to add some lines to prevent access to your OSC directories. There are several posts in the forums regarding this or I can send you mine if you get stuck.

 

Jack

Support Links:

For Hire: Contact me for anything you need help with for your shop: upgrading, hosting, repairs, code written, etc.

All of My Addons

Get the latest versions of my addons

Recommended SEO Addons

Link to comment
Share on other sites

Pretty much most of them. Here's a copy of the file that comes with the SEC distribution. Note that the wildcards used for the images is not valid for all SE's. You probably don't want to block images anyway. If you want to list with Froogle, access to your images will be needed.

 

Jack

 

# Robots.txt for http://www.yourdomain.com

# Email: [email protected]

#

#place this file in your root directory

#

#Replace all /catalog/ with your shop directory name

#Remove the # to activate additional blocks

#

#

User-agent: *

#

#Block All Images

Disallow: *.gif

Disallow: *.jpg

Disallow: /*.gif$

Disallow: /*.jpg$

#

#To block main page due to size from bots

# Disallow: /catalog/index.php

#

#To shut down site completely from bots use either

# Disallow: /

# Disallow: /catalog/

#

#If you had search engine friendly URLs on and want to clean them out of the bots

#Disallow: /catalog/default.php/cPath/

#

#

#Block out directories bots don't need to go to

Disallow: /catalog/admin/

Disallow: /catalog/download/

Disallow: /catalog/images/

Disallow: /catalog/includes/

Disallow: /catalog/pub/

#

#Block out things that are secure or login oriented

Disallow: /catalog/account.php

Disallow: /catalog/account_edit.php

Disallow: /catalog/account_edit_process.php

Disallow: /catalog/account_history.php

Disallow: /catalog/account_history_info.php

Disallow: /catalog/account_newsletters.php

Disallow: /catalog/account_notifications.php

Disallow: /catalog/account_password.php

Disallow: /catalog/add_checkout_success.php

Disallow: /catalog/address_book.php

Disallow: /catalog/address_book_process.php

Disallow: /catalog/addtocart.php

Disallow: /catalog/advanced_search.php

Disallow: /catalog/advanced_search_result.php

Disallow: /catalog/attributes_qty_prices_table_popup.php

Disallow: /catalog/checkout_confirmation.php

Disallow: /catalog/checkout_payment.php

Disallow: /catalog/checkout_payment_address.php

Disallow: /catalog/checkout_paypalipn.php

Disallow: /catalog/checkout_process.php

Disallow: /catalog/checkout_shipping.php

Disallow: /catalog/checkout_shipping_address.php

Disallow: /catalog/checkout_success.php

Disallow: /catalog/create_account.php

Disallow: /catalog/create_account_process.php

Disallow: /catalog/create_account_success.php

Disallow: /catalog/disclaimer.php

Disallow: /catalog/download.php

Disallow: /catalog/info_shopping_cart.php

Disallow: /catalog/login.php

Disallow: /catalog/logoff.php

Disallow: /catalog/password_forgotten.php

Disallow: /catalog/paypal_notify.php

Disallow: /catalog/popup_coupon_help.php

Disallow: /catalog/popup_image.php

Disallow: /catalog/popup_image1.php

Disallow: /catalog/popup_image2.php

Disallow: /catalog/popup_image3.php

Disallow: /catalog/popup_image4.php

Disallow: /catalog/popup_image5.php

Disallow: /catalog/popup_image6.php

Disallow: /catalog/popup_search_help.php

Disallow: /catalog/product_notifications.php

Disallow: /catalog/product_reviews_write.php

Disallow: /catalog/redirect.php

Disallow: /catalog/shopping_cart.php

Disallow: /catalog/shopping_cart_help.php

Disallow: /catalog/shipping_estimator_popup.php

Disallow: /catalog/tell_a_friend.php

#

#Optional Pages to block

#Disallow: /catalog/about_us.php

#Disallow: /catalog/conditions.php

#Disallow: /catalog/contact_us.php

#Disallow: /catalog/privacy.php

#Disallow: /catalog/shipping.php

Support Links:

For Hire: Contact me for anything you need help with for your shop: upgrading, hosting, repairs, code written, etc.

All of My Addons

Get the latest versions of my addons

Recommended SEO Addons

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...