Jump to content
  • Checkout
  • Login
  • Get in touch

osCommerce

The e-commerce.

Inktomi Spider


pjones

Recommended Posts

I'm running MS1 with the SID Killer. This works fine with Googlebot, but now I'm being relentlessly spidered by Inktomi. When I look I see SID's are present on all the pages it is spidering. Is there any way to deal with this?

Link to comment
Share on other sites

Would a robot.txt not do the job?

 

Ummn, better not do that. If you prohibit Inktoni from spidering your site, then they will simply not list you in their index.

 

I suppose this might be a good idea if you don't care about traffic.

-------------------------------------------------------------------------------------------------------------------------

NOTE: As of Oct 2006, I'm not as active in this forum as I used to be, but I still work with osC quite a bit.

If you have a question about any of my posts here, your best bet is to contact me though either Email or PM in my profile, and I'll be happy to help.

Link to comment
Share on other sites

Is there any technique to kill the SID for Inktomi?

Put the User Agent (UA) into the appropriate place. If you are using the Spider Killer commonly known as "burts" then simply amend includes/functions/html_output.php

 

If you are using MS2 (of which some versions have Spider Kill built in), then add the UA into the text file (I cannot remember the text file name).

 

I can 100% guarantee that if you are using "burts" and add the correct UA into it, it will strip the SID created by Inktomi. While you are there also add yahooseeker to the list.

 

Problems arise if your Oscommerce site is based upon a version commonly known as "loaded". If you are running a loaded version please see the appropriate forum for it (try searching for phesis in Google), as "loaded" help is not allowed to be given within the Oscommerce Forums.

Link to comment
Share on other sites

Burt,

 

I am running MS2 2.2 I. I just sent you a PM because i found this message through a search and should've checked the date. In any case people may benefit from the info you can share. I have a robots.txt file in the catalog/inlcudes directory. This file came with the the OSC version i downloaded. No where in any file is this robots.txt file referenced (ie: html_output.php or any other) as well this robots.txt file on it's own is invalid according to this robots.txt validator:

 

http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

 

I am thoroughly confused on what to do. I want to install your SID killer but apparently this functionality is built into MS2 2.2 .. It does not appear to be working for me though. Any ideas, comments are appreciated.

 

thanks!

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...