Jump to content
  • Checkout
  • Login
  • Get in touch

osCommerce

The e-commerce.

Robots.txt Permissons


Tim.Ford

Recommended Posts

Posted

Morning All,

 

I have been adjusting my robots.txt to include all agents and deny the robots to some areas. My main concern is that you can actaully type in the web address and include robots.txt at the end and view the file (www.awebsite.com/robots.txt). Is this correct?

 

Or should this not be viewable to the public. Or can i put in a redirect for any request to redirect to 404 or would these effect the robots??

 

Any help would be greatfully appreciated.

 

Thanks, tim.

Posted

Hi Jack,

 

Thanks for that that answered my question straight away. I was just conecerd about the files that i listed etc.

 

Thansk, Tim,

Posted
The basic robots.txt file has to be in the root. You can hide the directories but it is not considered a good idea as mentioned on this page.

 

Jack

 

besides, any browser can pose as a genuine robot and read it anyway, so securing robots.txt so that only robots can access it is futile and totally unnecessary.

Treasurer MFC

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...