Tim.Ford Posted March 21, 2006 Posted March 21, 2006 Morning All, I have been adjusting my robots.txt to include all agents and deny the robots to some areas. My main concern is that you can actaully type in the web address and include robots.txt at the end and view the file (www.awebsite.com/robots.txt). Is this correct? Or should this not be viewable to the public. Or can i put in a redirect for any request to redirect to 404 or would these effect the robots?? Any help would be greatfully appreciated. Thanks, tim.
Jack_mcs Posted March 21, 2006 Posted March 21, 2006 The basic robots.txt file has to be in the root. You can hide the directories but it is not considered a good idea as mentioned on this page. Jack Support Links: For Hire: Contact me for anything you need help with for your shop: upgrading, hosting, repairs, code written, etc. All of My Addons Get the latest versions of my addons Recommended SEO Addons
Tim.Ford Posted March 21, 2006 Author Posted March 21, 2006 Hi Jack, Thanks for that that answered my question straight away. I was just conecerd about the files that i listed etc. Thansk, Tim,
boxtel Posted March 22, 2006 Posted March 22, 2006 The basic robots.txt file has to be in the root. You can hide the directories but it is not considered a good idea as mentioned on this page. Jack besides, any browser can pose as a genuine robot and read it anyway, so securing robots.txt so that only robots can access it is futile and totally unnecessary. Treasurer MFC
Recommended Posts
Archived
This topic is now archived and is closed to further replies.