Jump to content
  • Checkout
  • Login
  • Get in touch

osCommerce

The e-commerce.

can someone tell me what to put in robots.txt


hpqnet

Recommended Posts

Hi, I have been slammed since day one on my bandwidth. I am using about 3GB a month, and I have got to do something about it. I did have to upgrade to 2GB a month for an extra $5 a month, but this also doubled my disk storage, so it was not so bad. I do not have a robots.txt file on my site and I am assuming this would help cut down on some of the bandwidth usage, right?

 

Can someone help?

 

I am using osc2.2ms2 and I have my catalog setup under / instead of the /catalog

My live site

Link to comment
Share on other sites

Hi

 

A robots text file will only help if your bandwidth is being used up by robots, which I assume it is, or why would you ask!?

 

All the information you need can be found here http://www.robotstxt.org/wc/exclusion.html

 

The difficult bit is in identifying the "un friendly" robots and excluding them [it is an exclusion protocol after all :)]

 

I have built up a long list of unwelcome robots over the last few months, and still add to it on a regular basis. If you would like a copy of my robots text file PM me and i will send you it with pleasure.

 

Hope this makes sense, I've just had a bottle of wine ;)

 

Regards

 

Spook

Link to comment
Share on other sites

Hi,

 

Your site looks good, the images and thumbnails were reasonable

 

Hi,

 

You could save a little bandwidth by changing your gif logos and thumbnails to jpg with about 40% compression.

 

The robots.txt file allows you to keep spiders out of certain directories, but this wont affect your bandwitdh much because spiders cant see images so they dont request them. A spider will see 10k html vs a customer will see 10k html +35k gif+15k gif...

 

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

Disallow: /catalog/admin/

Disallow: /catalog/images/

Disallow: /catalog/includes/

 

Basically when your bandwidth starts going up like that it means your site is becoming sucessfull, and it might be time to look at a host that can handle 25+gig/month bandwidth.

 

Dave...

Link to comment
Share on other sites

Thank you, should I also include /includes/ in the robots.txt file? Since I have my catalog under the / directory would this be good?

 

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/ <-- i was not sure about this one?

Disallow: /admin/

Disallow: /images/

Disallow: /includes/ <-- i was not sure about this one?

 

 

Regarding the gif vs. jpg, I though the gif files were smaller and loaded quicker than the same quality jpg files.. Is this not correct?

Link to comment
Share on other sites

Regarding the gif vs. jpg, I though the gif files were smaller and loaded quicker than the same quality jpg files.. Is this not correct?

 

Sometimes yes...sometimes no.

For thumbnails...no.

You cannot truly thumbnail a gif. A "thumbnailed" gif is basically just a smaller version of the larger gif, without the compression. A "thumbnailed" gif merely resizes the original without resampling it, so a 200 x 200 gif of 15Kb, resized to a 75 x 75 gif, will still be 15 Kb, whereas a jpg of the same dimension, when thumbnailed will reduce to a file size of about 2Kb.

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...