Jump to content
  • Checkout
  • Login
  • Get in touch


The e-commerce.

More Bot Crawlers!!


Recommended Posts

HI Everyone! Google is currently crawling my site around 10 - 15 times/day ; Yahoo 5-10 / day ; and MSN 5-10/day. Is this good? What can I do to increase it. I have a spider txt file created and have the SEO contrib installed....any suggestions??


Here is a Snapshot of my report from yesterday: not a bad bot day..


Rank Browser Number of hits

1 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 24

2 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) 13

3 (+http://search.msn.com/msnbot.htm) 11





Link to comment
Share on other sites

there is no spider.txt file... there is a robots.txt file.


Well as far as google is concerned, you don't need much more than that.


Yahoo on the other hand has hit me 4,100 time this month.... no idea why it's soo insane or anything.


I use awstats so I don't quite know my daily um.. limits.... except the simple stuff.

Link to comment
Share on other sites

There is a spiders.txt file! It is used to prevent spiders using sessions so very important.


stevel regularly updates the spiders.txt, big up Steve! You can get it from here:





I actually did mean Robots.txt, but am glad to have the spider.txt info as well.


Anyone else have suggestions on how to increase SEO with or BOT activity?



Link to comment
Share on other sites


This topic is now archived and is closed to further replies.

  • Create New...