Spider Destroying Site!?

[Guest]

Ok, the last couple of days my website has been very slow or I cannot log onto the main page or my admin I get "Cannot Find Server".

 

When I do get on I can see in my User tracking that "msnbot64130.search.msn.com" is online and has currently 241 clicks and rising. It has been on for 23:50:18. The thing is this goes up to about 23:55 then will go back down to 23:40 but the clicks are still rising.

 

Is this spider using up all my bandwidth and stopping me and my customers getting online? Or is it something else?

 

I have searched these forums asked my hosts with no luck.

 

I am just about to try and ban the IP address using .htaccess hoping this will work. But I cannot get onto my control panel through my host as this also says "Cannot find server"

 

So far my host (Doteasy) have not got back to me. Normally they are very good.

 

I would appreciate any help from anybody.

 

Many Thanks in advance.

 

Also I better add it is NOT creating any sessions. And I do have prevent spider sessions to True.

[stevel]

msnbot is not in the default spiders.txt list. Add it.

[Guest]

Thanks for the reply.

 

I already have msnbot added in my spiders list. They are not creating sessions.

 

I can see the bot as I have user tracking installed. These shows me they are constantly online.

 

Last night when the bot wasnt there it seemed to load fine. So I tried banning the IP using .htaccess, but this morning it wont load again and I cant get on to see if the bot is back on.

[stevel]

How is it causing you trouble?

 

I suggest you go to http://search.msn.com/msnbot.htm and see if there is something there about asking them not to spider your site so much, if it's a problem.

[SGershon]

From the above link:

# How do I prevent MSNBot from crawling some or all of my website?

The robots.txt file is used to prevent web crawlers from accessing a web site. The format of the robots.txt file is specified in The Robot Exclusion Standard. MSNBot analyzes all instances where the User-Agent is specified as either "msnbot" or "*". Based on this, MSNBot crawls only the web pages that allow it to do so.

 

# Why is MSNBot disobeying my robots.txt file?

MSNBot downloads the robots.txt file once a day, so it may take a few moments for MSNBot to learn about changes you make. Ensure that the syntax of your robots.txt file is correct according to the standard at http://www.robotstxt.org/wc/exclusion.html#robotstxt. Also, make sure the robots.txt file is in your server's top directory. If it is in a subdirectory, it will not affect on how MSNBot crawls your web site.

 

# How do I report problems to MSN Search?

If you have questions about our MSNBot technology, please e-mail them to [email protected]. In your e-mail message, include as much detail as possible about the problem you are experiencing. Thank you!

[Guest]

To be honest I was not sure if it was the spider that has been causing my website problems. I only notcied when it wasnt there it ran fine. (Maybe coincidence)

 

At moment I have had no luck getting back onto my site at all after editing the .htaccess so not sure what to do.

[Guest]

Ok, im desperate for some help. My site has been up and down for the last three days. Everything was fine before then.

 

The thing is I cannot login into my hosts site admin panel to edit .htaccess or any files. Or I cannot login using CuteFTP. Does this sound like something that should happen because of a spider on the site? Or is this more to do with the server?

 

I have contacted my host a number of times who are normally extremely good. But I have had no reply apart from the first email I sent where they said it looked fine and it was probably my internet connection.

 

Any help/advice would be appreciated.

[stevel]

Your not being able to access by FTP has nothing to do with the spiders. Contact your host's support desk for help with that.

[roby2004]

change your host. If your host cant handle a simple spider how would they handle a vast bandwidth?