Guest Posted July 6, 2006 Posted July 6, 2006 If I have checkout_shipping.php listed in the robot.txt file as disallow, how can MSNbot be crawling the page? I just checked my "Who's Online" section via Admin and used a IP tracer to see who it was (praying it was a customer LOL), and it said MSNBot.
choosealogin Posted July 6, 2006 Posted July 6, 2006 There's one thing I read somewhere, I thought it was MSIEcrawler, though...anyway, it said it would show up as a bot (msiecrawler) when someone uses that 'download and view offline' function...or whatever it's called, in IE. But that wouldn't make sense in this case, since you used an ip tracer. Hmm. Sorry.
Guest Posted July 6, 2006 Posted July 6, 2006 No worries Mark. Since I posted this thread, I've done searches for robot.txt on this forum. I was looking for an up-to-date suggested file. I have no clue if mine has enough information in it. I did just update my spiders.txt as per stevel's contribution of July 7, 2006. I'm thinking of installing Jack's Who's Online Enhancement. That way I won't have to do IP tracing separately. I do have Add Free Stats on my .html pages which gives all that info. I wonder if I can put the coding into the store's php pages? *sighs* I'm so kunphoozled! LOL
New 
Recommended Posts
Archived
This topic is now archived and is closed to further replies.