Jump to content
  • Checkout
  • Login
  • Get in touch


The e-commerce.

Why are spiders bad?


Recommended Posts

Please excuse my ignorance, but why would one choose to kill spiders? I thought spiders were a good thing to get your site listed.


I have spent hours reading various posts about SEO - whether to use or not to use the freindly URLs and the like.


Can someone please point me in the right direction. I have read jackof's nice posts about killing spiders and using Linda's headers, but really don't understand the harm in spiders.


Any insight here would be greatly appreciated.


thanking you in advance,


George Shirey

Link to comment
Share on other sites

The problem with the PHP sessions is that, by default, the code will try to set a cookie storing the session number. Because you cannot set a cookie for a spider, the SID is passed in the URL instead. However, the sID is different for every session.


Spiders like to revisit pages to see if the content has changed, etc. It can take several spider visits to the same page for it to be properly indexed. However, when the sID is appended to the URL , the spider can never find the same URL twice (because the sID has changed). This can kill a site's search-engine rankings.


The session-killer simply removes the sID from the URL for known spiders. It does not stop any pages from being indexed; robot.txt does that. In-fact, it does the opposite, and helps your pages be spidered.

Link to comment
Share on other sites

Now my question is: I have spent tons of time in the forums reading about the stuff and am trying to figure out what the best way to kill the SIDs is.


Is that what "Burts" mod is for? If so, what's the difference between a mod and a contribution?


Also, if I am forcing cookie usage shouldn't that make the matter obsolete? But then again that raises the question for me, if I'm forcing cookie usage, am I dissalowing my pages to be crawled?


How deep does this tunnel go?

Link to comment
Share on other sites

MS2 integrates a spider killer, so if you're using this release, there is no need to add a contribution.


Yes, forcing cookie use will solve the SID/spider issue, but also make your site unuseable for anyone browsing with cookies disabled.


It's a choice you have to make.

Link to comment
Share on other sites


This topic is now archived and is closed to further replies.

  • Create New...