Jump to content
  • Checkout
  • Login
  • Get in touch

osCommerce

The e-commerce.

How to write a robots.txt file???


tkm

Recommended Posts

I looked at one site that recommended writing a robots.txt file like this

 

# All robots will spider the domain

User-agent: *

Disallow:

 

# Disallow directory cgi-bin

User-agent: *

Disallow: /cgi-bin/

 

But another site recommended this

# All robots will spider the domain

User-agent: *

Disallow:

 

# Disallow directory cgi-bin

User-agent: *

Disallow: cgi-bin

 

And said that if you use "/" the search engines would ban you.

 

Which is the correct method?

Link to comment
Share on other sites

  • 3 months later...

I hope so one can give me a solution for the ia_archiver robot. It is still continuing to spider my entire site even though all the other robots are obeying the robots.txt file and not going past the /catalog/ that I have indicated.

 

I even listed them separately and they are still going everywhere:

User agent: ia_archiver

Disallow: /catalog/

 

I'm also using this contribution: http://www.oscommerce.com/community/contributions,1089

 

Is this the correct contribution to use? Threads state don't use Ians, use Burts, but with all the different threads now its hard to tell which one is which anymore.

 

Thanks in advance!

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...