pstrid Posted January 16, 2005 Posted January 16, 2005 Some search engines are still getting SIDs and I am not sure why. Here is my Inktomi/Yahoo's refferer log: 68.142.251.21 - - [13/Jan/2005:20:23:17 -0600] GET /product_info.php?products_id=73&osCsid=fb13c970791f1f8a1fb1fe344c05ccd3 HTTP/1.0 200 2327 - Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) 68.142.251.21 - - [16/Jan/2005:10:02:54 -0600] GET /shop.php?osCsid=858223984b7d3b0f65ab5820abfcd6d0 HTTP/1.0 200 1702 - Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) I have 'Prevent Spider Sessions' set to true. And I have the following in my spider.txt file: inktomisearch inktomisearch.com lj1029.inktomisearch.com slurp slurp/si [email protected] Yahoo! Slurp yahoo! slurp yahoobot yahooseeker yahooseeker/1.1 Yahoo-MMCrawler/3.x yahoo-mmcrawler/3.x Anyone have any ideas why they would still be getting SIDs? Other spiders are not generating SIDs so it looks like it is working for some search engines but not this one :( Thanks for you help.
Debs Posted January 16, 2005 Posted January 16, 2005 Some search engines are still getting SIDs and I am not sure why. Here is my Inktomi/Yahoo's refferer log: 68.142.251.21 - - [13/Jan/2005:20:23:17 -0600] ?GET /product_info.php?products_id=73&osCsid=fb13c970791f1f8a1fb1fe344c05ccd3 HTTP/1.0 ?200 2327 ?- ? Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) 68.142.251.21 - - [16/Jan/2005:10:02:54 -0600] ?GET /shop.php?osCsid=858223984b7d3b0f65ab5820abfcd6d0 HTTP/1.0 ?200 1702 ?- ? Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) I have 'Prevent Spider Sessions' set to true. And I have the following in my spider.txt file: inktomisearch inktomisearch.com lj1029.inktomisearch.com slurp slurp/si [email protected] Yahoo! Slurp yahoo! slurp yahoobot yahooseeker yahooseeker/1.1 Yahoo-MMCrawler/3.x yahoo-mmcrawler/3.x Anyone have any ideas why they would still be getting SIDs? Other spiders are not generating SIDs so it looks like it is working for some search engines but not this one :( Thanks for you help. <{POST_SNAPBACK}> I added Mozilla/5.0 as the user agent. I fought this same problem for the last 1 1/2 years... all the experts will tell you that it's just an old sid getting respidered... not true! Adding Mozilla/5.0 has helped, but not completely cured the yahoo-slurp crawler from adding sids. I have spent weeks reading every post and nobody seems to know how to solve it. Most seem unaware that they have the same problem.. but after doing a search on thier own website... they all seem to have sids too (in yahoo). Adding Mozilla/5.0 as the user agent has helped me the most. Good luck
♥Vger Posted January 16, 2005 Posted January 16, 2005 I knew someone with this problem, and it was resolved by just adding 'inktomi' (without the quotes). Vger
pstrid Posted January 16, 2005 Author Posted January 16, 2005 Thanks, i'll add both of these and see how i make out. Also, one more question... are entries on spider.txt case sensitive? I have heard different responses on this topic.
New 
Recommended Posts
Archived
This topic is now archived and is closed to further replies.