/robots.txt <--- what should be there?

[agiftcodotcom]

When I view my error reports, I see that someone (or something/bot) is getting a 404 when looking for /robots.txt

 

That's because I don't have a /robots.txt

 

What should I have for /robots.txt?

 

Any help is appreciated.

[stevel]

You don't need one. But if you don't want the 404s, create one such as this one line:

 

User-agent: *

 

On the other hand, it's a good idea to keep robots out of pages such as the shopping cart, etc. Here's what I have:

User-agent: *
Disallow: /shopping_cart.php
Disallow: /advanced_search.php
Disallow: /login.php
Disallow: /checkout_shipping.php
Disallow: /account.php
Disallow: /login.php
Disallow: /create_account.php
Disallow: /password_forgotten.php

[peterr]

Hi,

 

I've read on many search engine optimisation forums that even though you can specify what files/paths are 'allowed', most bots/spiders don't follow the 'rules' anyway, it won't stop them spidering a path/file after all. :D

 

Peter

[agiftcodotcom]

So basically... not needed. Cool, thanks guys.

[peterr]

Eric,

 

As Steve said, just create a file called robots.txt with this line in it

 

User-agent: *

 

Place it in your web root path (ususally called public_html).

 

Then that gets rid of the 404 messages.

 

Peter

[stevel]

I've read on many search engine optimisation forums that even though you can specify what files/paths are 'allowed', most bots/spiders don't follow the 'rules' anyway, it won't stop them spidering a path/file after all.

That doesn't seem to be the case from what I can tell. Since I added the robots.txt file as above, none of the search engines have gone down the disallowed paths. It certainly doesn't hurt in any case.

[agiftcodotcom]

Added the file, thanks :)

[burt]

http://www.robotstxt.org is a good resource.

[agiftcodotcom]

Thanks :D