Jump to content
  • Checkout
  • Login
  • Get in touch

osCommerce

The e-commerce.

Inktomi heavily slurped


Tomcat

Recommended Posts

Hi all,

 

I get "slurped" by Inktomi seveal times a day, about 1600 hits in two months...

 

Does anybody know how to get rid of inktomi spider ( and most of all if I SHOULD get rid of it ) ? May be with a robots.txt file ?

 

Is it really usefull to get spidered so many times by inktomi ?

 

Thanks

Franco

Outside links in signatures are not allowed!

Link to comment
Share on other sites

No, you should not get rid of it.

 

Just make sure that the URLs that it is spidering do not have SIDs on them.

-------------------------------------------------------------------------------------------------------------------------

NOTE: As of Oct 2006, I'm not as active in this forum as I used to be, but I still work with osC quite a bit.

If you have a question about any of my posts here, your best bet is to contact me though either Email or PM in my profile, and I'll be happy to help.

Link to comment
Share on other sites

Chris,

 

How can I be sure if Inktomi gets session IDs ?

 

I have Burt's Sid killer and I have already put Inktomi on the list of spiders

 

Franco

Outside links in signatures are not allowed!

Link to comment
Share on other sites

Well, the easiest way would be to look in your whos_online section of admin when the inktomi bot is there, and visually scan the ip addresses that belong to inktomi. You should not see any SIDs for those urls.

 

Another way would be to go to your favorite spider simulator, and use it to spider your site. Right after kicking off a spidering session from the sim, check your logs for the auser agent of that simulator, and add it to the list of user agents in your spider killer. hen spider your site again using the spider sim, and see if it picks up any SIDs. It should not.

-------------------------------------------------------------------------------------------------------------------------

NOTE: As of Oct 2006, I'm not as active in this forum as I used to be, but I still work with osC quite a bit.

If you have a question about any of my posts here, your best bet is to contact me though either Email or PM in my profile, and I'll be happy to help.

Link to comment
Share on other sites

Right,

 

I did the search in the whos_online, now can I add the IP address straight to the list ?

Inktomi has many IPs some are already blocked from getting session while others aren't

Outside links in signatures are not allowed!

Link to comment
Share on other sites

More info.

 

http://www.oscommerce.com/forums/viewtopic.php?t=36577

http://www.oscommerce.com/forums/viewtopic.php?t=39566

 

You must be confusted, becaues Burt's spider Killer doesn't use IP addresses, it uses the user agent string.

 

None of the URLs that Inktomi spiders should have SIDs, otherwise Inktomi will get caught in an endless loop, and will continue to spider your site all the time. I've seen this bot do over 60,000 hits in less than a month, and suck up over 2 gigs of bandwidth when it gets stuck in this loop.

 

I'd highly recommend reading every thread you can find regarding search engines and SIDs.

-------------------------------------------------------------------------------------------------------------------------

NOTE: As of Oct 2006, I'm not as active in this forum as I used to be, but I still work with osC quite a bit.

If you have a question about any of my posts here, your best bet is to contact me though either Email or PM in my profile, and I'll be happy to help.

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...