robots.txt instructions for MSN bots

[♥Biancoblu]

Hi,

I have read several oscommerce threads about robots.txt but couldn't find an answer to my question yet:

I read on some sites that seeing as MSN don't have a place to submit a sitemap the way you do at Google webmaster tools and Yahoo site explorer, some people suggest to add an instruction in robots.txt to kind of guide MSN bots to your sitemap.

So they suggest to do the following in robots.txt:

 

User-agent: *

Sitemap: http://www.yourdomain.com/sitemap.xml

Disallow: /put here all pages/folders you don't want crawled

 

This way, apparently, when MSN bots get to your robots.txt, they are told to crawl your sitemap.xml file whilst ignoring your disallowed list.

 

Do you guys think it's a good idea?

 

Isabella

[Guest]

Hi,

I have read several oscommerce threads about robots.txt but couldn't find an answer to my question yet:

I read on some sites that seeing as MSN don't have a place to submit a sitemap the way you do at Google webmaster tools and Yahoo site explorer, some people suggest to add an instruction in robots.txt to kind of guide MSN bots to your sitemap.

So they suggest to do the following in robots.txt:

This way, apparently, when MSN bots get to your robots.txt, they are told to crawl your sitemap.xml file whilst ignoring your disallowed list.

 

Do you guys think it's a good idea?

 

Isabella

 

yes. a good robots.txt file is suggested. make sure to point to your sitemap in your robots.txt file

[♥Biancoblu]

Thanks for replying. :)