Jump to content
  • Checkout
  • Login
  • Get in touch

osCommerce

The e-commerce.

what exactly does spiders.txt.do?


ollyno1uk2

Recommended Posts

Hi there

 

I have both spiders.txt and robots.txt on my site. I guess robots.txt just restricts access to pages you dont want listed in the search engines right?

 

With spiders.txt how important is it? will spiders not get onto my page unless they are listed? How important is it to keep this file updated?

 

Thanks a lot

Link to comment
Share on other sites

Hi there

 

I have both spiders.txt and robots.txt on my site. I guess robots.txt just restricts access to pages you dont want listed in the search engines right?

 

With spiders.txt how important is it? will spiders not get onto my page unless they are listed? How important is it to keep this file updated?

 

Thanks a lot

 

I think it could be very importent for different reasons;

 

Your ranking will suffer without it, you won't be able restrict other bots that could eat your bandwidth and you could get things like you admin files all over the search engines. I may be wrong but having one is much better the not IMO.

-----------------------------------------------------------------------------

OSC user for years and no coder, so I've earned my stripes.

 

Feel free to private message me.

Link to comment
Share on other sites

I think it could be very importent for different reasons;

 

Your ranking will suffer without it, you won't be able restrict other bots that could eat your bandwidth and you could get things like you admin files all over the search engines. I may be wrong but having one is much better the not IMO.

 

robots.txt does not restrict anything, it merely states your preference as to where you would like spiders not to index. It is up to the spider to honor that preference or simply ignore it. robots.txt is an internet wide standard inplementation.

 

spiders.txt is a list of know spider user-agents. It is used to identify spiders on your site so that they are not issued session id's. spiders.txt is osc specific implementation.

Treasurer MFC

Link to comment
Share on other sites

robots.txt does not restrict anything, it merely states your preference as to where you would like spiders not to index. It is up to the spider to honor that preference or simply ignore it. robots.txt is an internet wide standard inplementation.

 

spiders.txt is a list of know spider user-agents. It is used to identify spiders on your site so that they are not issued session id's. spiders.txt is osc specific implementation.

 

On that basis it WOULD restrict all bots that obay the standard so it is used to restrict isn't it.

 

But I'm no expert and anything i say on here is only my own opinion based on my experiance so for full details perhaps a professional is better all the time.

 

http://www.robotstxt.org

-----------------------------------------------------------------------------

OSC user for years and no coder, so I've earned my stripes.

 

Feel free to private message me.

Link to comment
Share on other sites

On that basis it WOULD restrict all bots that obay the standard so it is used to restrict isn't it.

 

But I'm no expert and anything i say on here is only my own opinion based on my experiance so for full details perhaps a professional is better all the time.

 

http://www.robotstxt.org

 

no it isn't

Treasurer MFC

Link to comment
Share on other sites

Hi there

 

I have both spiders.txt and robots.txt on my site. I guess robots.txt just restricts access to pages you dont want listed in the search engines right?

 

With spiders.txt how important is it? will spiders not get onto my page unless they are listed? How important is it to keep this file updated?

 

Thanks a lot

 

 

Well there is a script that compares the user agent with the text that is in the spider file to decide whether a spider is in or a visitor is in.

With that we canmake out whether to attach session id or not.

 

 

Satish

Ask/Skype for Free osCommerce value addon/SEO suggestion tips for your site.

 

Check My About US For who am I and what My company does.

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...