Jump to content
  • Checkout
  • Login
  • Get in touch


The e-commerce.

Bots getting stuck in my site! HELP!!!


Recommended Posts

Hi all!


I need your help!


It seems that Googlebot and MSNbot are both getting caught in my site. They are downloading over 300MB a day and its costing me heaps of money!


Can anyone help me figur out why they are getting stuck. I can't for the life of me figure it out.


Please help!!!!


In the mean time i have added a robots.txt to disallow goglebot and msnbot til the end of the month.


Thanks Edwin!

Link to comment
Share on other sites

Already did that but can find any relevant posts or a solution to this problem.


Please can some one help!


P.S. Thanks agiftcodotcom for your help so far.

Link to comment
Share on other sites

Im having a problem with my website and not sure if its to do with this too.


Basically my website the last 2/3 days has been slow or I get an error "Cannot find server" this is when logging onto my main page or admin.


When I do get on I see in my user tracking that an msn bot is searching through all my pages. It has no session but has been online for 23hours and is going through every page. msnbot64130.search.msn.com, IP:


Could this be the problem why my website is not loading??


If so how do I stop them from visiting or is there another way to fix it??


I have asked my host (doteasy) who are normally very good but have had no resonse.


Many Thanks for any help.

Link to comment
Share on other sites

I know this is cross posting on sites, but I'm not going to copy all that info over here. It's a lot of info.


On a side note I'm part of Gallery's dev people...and that is were I posted tons of info along with another moderator on that site.


What you want to do is create a "robots.txt" this keeps most of the web bots from accessing pages & folders that they don't need to be accessing and allowing them access to your pages that you want to be indexed for ranking on search engines.


This thread covers .htaccess Banning IP, Bad referer, Hotlinking, along with the Robot.txt







(I'm Jade Dragon on that site)

Link to comment
Share on other sites


This topic is now archived and is closed to further replies.

  • Create New...