hpqnet Posted February 28, 2004 Share Posted February 28, 2004 Hi, I have been slammed since day one on my bandwidth. I am using about 3GB a month, and I have got to do something about it. I did have to upgrade to 2GB a month for an extra $5 a month, but this also doubled my disk storage, so it was not so bad. I do not have a robots.txt file on my site and I am assuming this would help cut down on some of the bandwidth usage, right? Can someone help? I am using osc2.2ms2 and I have my catalog setup under / instead of the /catalog my live site Link to comment Share on other sites More sharing options...
Jack_mcs Posted February 28, 2004 Share Posted February 28, 2004 Take a look here -> Robots. It should answer all of your questions and provides an example of a robots.txt file too. You'll want to add some lines to prevent access to your OSC directories. There are several posts in the forums regarding this or I can send you mine if you get stuck. Jack Support Links: For Hire: Contact me for anything you need help with for your shop: upgrading, hosting, repairs, code written, etc. All of My Addons Get the latest versions of my addons Recommended SEO Addons Link to comment Share on other sites More sharing options...
hpqnet Posted February 28, 2004 Author Share Posted February 28, 2004 But what osc directories can I safely exclude? That is where I am stuck, I do not want to disallow some directories that may need to be allowed for proper indexing. Thanks Link to comment Share on other sites More sharing options...
Jack_mcs Posted March 2, 2004 Share Posted March 2, 2004 Pretty much most of them. Here's a copy of the file that comes with the SEC distribution. Note that the wildcards used for the images is not valid for all SE's. You probably don't want to block images anyway. If you want to list with Froogle, access to your images will be needed. Jack # Robots.txt for http://www.yourdomain.com # Email: [email protected] # #place this file in your root directory # #Replace all /catalog/ with your shop directory name #Remove the # to activate additional blocks # # User-agent: * # #Block All Images Disallow: *.gif Disallow: *.jpg Disallow: /*.gif$ Disallow: /*.jpg$ # #To block main page due to size from bots # Disallow: /catalog/index.php # #To shut down site completely from bots use either # Disallow: / # Disallow: /catalog/ # #If you had search engine friendly URLs on and want to clean them out of the bots #Disallow: /catalog/default.php/cPath/ # # #Block out directories bots don't need to go to Disallow: /catalog/admin/ Disallow: /catalog/download/ Disallow: /catalog/images/ Disallow: /catalog/includes/ Disallow: /catalog/pub/ # #Block out things that are secure or login oriented Disallow: /catalog/account.php Disallow: /catalog/account_edit.php Disallow: /catalog/account_edit_process.php Disallow: /catalog/account_history.php Disallow: /catalog/account_history_info.php Disallow: /catalog/account_newsletters.php Disallow: /catalog/account_notifications.php Disallow: /catalog/account_password.php Disallow: /catalog/add_checkout_success.php Disallow: /catalog/address_book.php Disallow: /catalog/address_book_process.php Disallow: /catalog/addtocart.php Disallow: /catalog/advanced_search.php Disallow: /catalog/advanced_search_result.php Disallow: /catalog/attributes_qty_prices_table_popup.php Disallow: /catalog/checkout_confirmation.php Disallow: /catalog/checkout_payment.php Disallow: /catalog/checkout_payment_address.php Disallow: /catalog/checkout_paypalipn.php Disallow: /catalog/checkout_process.php Disallow: /catalog/checkout_shipping.php Disallow: /catalog/checkout_shipping_address.php Disallow: /catalog/checkout_success.php Disallow: /catalog/create_account.php Disallow: /catalog/create_account_process.php Disallow: /catalog/create_account_success.php Disallow: /catalog/disclaimer.php Disallow: /catalog/download.php Disallow: /catalog/info_shopping_cart.php Disallow: /catalog/login.php Disallow: /catalog/logoff.php Disallow: /catalog/password_forgotten.php Disallow: /catalog/paypal_notify.php Disallow: /catalog/popup_coupon_help.php Disallow: /catalog/popup_image.php Disallow: /catalog/popup_image1.php Disallow: /catalog/popup_image2.php Disallow: /catalog/popup_image3.php Disallow: /catalog/popup_image4.php Disallow: /catalog/popup_image5.php Disallow: /catalog/popup_image6.php Disallow: /catalog/popup_search_help.php Disallow: /catalog/product_notifications.php Disallow: /catalog/product_reviews_write.php Disallow: /catalog/redirect.php Disallow: /catalog/shopping_cart.php Disallow: /catalog/shopping_cart_help.php Disallow: /catalog/shipping_estimator_popup.php Disallow: /catalog/tell_a_friend.php # #Optional Pages to block #Disallow: /catalog/about_us.php #Disallow: /catalog/conditions.php #Disallow: /catalog/contact_us.php #Disallow: /catalog/privacy.php #Disallow: /catalog/shipping.php Support Links: For Hire: Contact me for anything you need help with for your shop: upgrading, hosting, repairs, code written, etc. All of My Addons Get the latest versions of my addons Recommended SEO Addons Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.