Stop Msn.bot indexing Site

[gaspower]

Hello,

 

we have had our site up and running for many years, version prior to MS2.31. Just recently we are having a msn.bot indexing our site and it is absolutely crippling our site. The site almost comes to a complete holt, and we have to wait until the indexing is over. Is there anyway to prevent this from msn.bot? We have other search engines on the site, and no issues.

 

Thanks JR

[Jack_mcs]

I've noticed they seem to be increasing their searches too. But if you stop them altogether, you would prevent your site from being listed by them, which is not good. You can add the following to your robots file and it should help. Change them as you wish, of course.

Crawl-delay: 2
Request-rate: 1/2
Visit-time: 0300-0900

[gaspower]

Thank you very much.

[gaspower]

Hello,

 

I should of asked. Do I need to use the robot addon contribution?

 

JR

[Jack_mcs]

I don't know what that is.

[gaspower]

Hello,

I was able to understand. I created a file called http://www.domain.com/robots.txt with below content,

 

Crawl-delay: 2

Request-rate: 1/2

Visit-time: 0300-0900

 

I set permissions to 644, correct?

 

Thanks JR

[Jack_mcs]

You should have a user agent statement. see http://www.robotstxt.org/ for examples and testing. The permissions don't matter but shouldn't be higher than 644.

[gaspower]

Hello,

 

Got it,

 

User-agent: msnbot

Crawl-delay: 20

Request-rate: 1/2

Visit-time: 0300-0900

 

Thanks JR