Jump to content
  • Checkout
  • Login
  • Get in touch


The e-commerce.

Spider Destroying Site!?


Recommended Posts

Ok, the last couple of days my website has been very slow or I cannot log onto the main page or my admin I get "Cannot Find Server".


When I do get on I can see in my User tracking that "msnbot64130.search.msn.com" is online and has currently 241 clicks and rising. It has been on for 23:50:18. The thing is this goes up to about 23:55 then will go back down to 23:40 but the clicks are still rising.


Is this spider using up all my bandwidth and stopping me and my customers getting online? Or is it something else?


I have searched these forums asked my hosts with no luck.


I am just about to try and ban the IP address using .htaccess hoping this will work. But I cannot get onto my control panel through my host as this also says "Cannot find server"


So far my host (Doteasy) have not got back to me. Normally they are very good.


I would appreciate any help from anybody.


Many Thanks in advance.


Also I better add it is NOT creating any sessions. And I do have prevent spider sessions to True.

Link to comment
Share on other sites

Thanks for the reply.


I already have msnbot added in my spiders list. They are not creating sessions.


I can see the bot as I have user tracking installed. These shows me they are constantly online.


Last night when the bot wasnt there it seemed to load fine. So I tried banning the IP using .htaccess, but this morning it wont load again and I cant get on to see if the bot is back on.

Link to comment
Share on other sites

How is it causing you trouble?


I suggest you go to http://search.msn.com/msnbot.htm and see if there is something there about asking them not to spider your site so much, if it's a problem.

Link to comment
Share on other sites

From the above link:

# How do I prevent MSNBot from crawling some or all of my website?

The robots.txt file is used to prevent web crawlers from accessing a web site. The format of the robots.txt file is specified in The Robot Exclusion Standard. MSNBot analyzes all instances where the User-Agent is specified as either "msnbot" or "*". Based on this, MSNBot crawls only the web pages that allow it to do so.


# Why is MSNBot disobeying my robots.txt file?

MSNBot downloads the robots.txt file once a day, so it may take a few moments for MSNBot to learn about changes you make. Ensure that the syntax of your robots.txt file is correct according to the standard at http://www.robotstxt.org/wc/exclusion.html#robotstxt. Also, make sure the robots.txt file is in your server's top directory. If it is in a subdirectory, it will not affect on how MSNBot crawls your web site.


# How do I report problems to MSN Search?

If you have questions about our MSNBot technology, please e-mail them to [email protected]. In your e-mail message, include as much detail as possible about the problem you are experiencing. Thank you!

If at first you don't succeed, you must be a programmer.


Tip Posted: Languageless Reviews

Link to comment
Share on other sites

To be honest I was not sure if it was the spider that has been causing my website problems. I only notcied when it wasnt there it ran fine. (Maybe coincidence)


At moment I have had no luck getting back onto my site at all after editing the .htaccess so not sure what to do.

Link to comment
Share on other sites

Ok, im desperate for some help. My site has been up and down for the last three days. Everything was fine before then.


The thing is I cannot login into my hosts site admin panel to edit .htaccess or any files. Or I cannot login using CuteFTP. Does this sound like something that should happen because of a spider on the site? Or is this more to do with the server?


I have contacted my host a number of times who are normally extremely good. But I have had no reply apart from the first email I sent where they said it looked fine and it was probably my internet connection.


Any help/advice would be appreciated.

Link to comment
Share on other sites


This topic is now archived and is closed to further replies.

  • Create New...