Jump to content
  • Checkout
  • Login
  • Get in touch

osCommerce

The e-commerce.

Robots.txt permission setting and spidering


smurk75

Recommended Posts

Below is a copy of my awstats spider report. What is confusing me is how come I have all these spider crawls but no 'Successful Hits' denoted by numbers after the "+" Currently I have the permissions set for robots.txt at "640" so others cannot read my file. What is the correct permission?

 

Googlebot 202

Inktomi Slurp 13

Unknown robot (identified by hit on 'robots.txt') 0+3

MSNBot 1

Unknown robot (identified by 'crawl') 1

 

* Robots shown here gave hits or traffic "not viewed" by visitors, so they are not included in other charts. Numbers after + are successful hits on "robots.txt" files

 

 

robots.txt

User-agent: Googlebot-Image

Disallow: /store/images/

 

User-agent: *

Disallow: /cgi-bin/

Disallow: /store/account.php

Disallow: /store/advanced_search.php

Disallow: /store/checkout_shipping.php

Disallow: /store/create_account.php

Disallow: /store/login.php

Disallow: /store/login.php

Disallow: /store/password_forgotten.php

Disallow: /store/shopping_cart.php

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...