danzie1 Posted September 10, 2005 Share Posted September 10, 2005 im doing a robots.txt file, and i have copied the file to the catalog directory (where my index is located), i then ran robots validator to the file and it was Validated, with no errors or problems. However when i then run META TAG ANALYZER at http://www.scrubtheweb.com/cgi-bin/webtools/meta-check.cgi i am told Note! This site does NOT provide the robots.txt file or the robots.txt file is not constructed properly. It is recommended all sites provide this file. For more information on how to properly construct and use the robots.txt file any ideas? Also should i see the word robots.txt if i go view - source on my index page? - cause i can see it Link to comment Share on other sites More sharing options...
Jan Zonjee Posted September 10, 2005 Share Posted September 10, 2005 i have copied the file to the catalog directory Wrong, your robots.txt file is looked for by spiders in the root directory not anywhere else.Also should i see the word robots.txt if i go view - source on my index page? No, it is just a text file for the convenience of the spiders. If you type in the url of that file (www.someoscshop/robots.txt) you will see the file in your browser window because it will be treated like any other .txt file. Usually it will be displayed in something like Courier. Link to comment Share on other sites More sharing options...
danzie1 Posted September 10, 2005 Author Share Posted September 10, 2005 but catalog is my root directory, isnt it as this is the site i will be submitting to the search engines, i dont have a redirect from www.mysite.com to my catalog everything exists on my website as www.mysite.com/catalog Link to comment Share on other sites More sharing options...
danzie1 Posted September 10, 2005 Author Share Posted September 10, 2005 but catalog is my root directory, isnt it as this is the site i will be submitting to the search engines, i dont have a redirect from www.mysite.com to my catalog everything exists on my website as www.mysite.com/catalog <{POST_SNAPBACK}> just to add to this if you go to just www.mysite.com you get....... Index of / Name Last modified Size Description -------------------------------------------------------------------------------- Parent Directory 10-Sep-2005 01:11 - catalog/ 10-Sep-2005 02:50 - cgi-bin/ 02-Sep-2005 03:43 - robots.txt 10-Sep-2005 02:22 1k so would i just submit my address www.mysite.com/catalog to the search engines? Also would u suggest i make some kind of re-direct to catalog, if people type in just www.mysite.com ?? If so how? Link to comment Share on other sites More sharing options...
Jan Zonjee Posted September 10, 2005 Share Posted September 10, 2005 Also would u suggest i make some kind of re-direct to catalog, if people type in just www.mysite.com ?? If so how? Personally, I would move all files and folders in the catalog directory (with the exception of the admin folder) to the root directory and change the two configure.php files accordingly.Then I would rename the folder catalog to something non-obvious and the name of the admin folder to something non-obvious (like adm_1n) and make sure you change the .htacccess file accordingly to these changes (you wouldn't be able to get into your own admin again). The first change is better for spiders, the second to stop hack attempts. If they can't find your admin folder, how can they try to get in? If you are not in for that, you at least should redirect everybody and everything that goes to mysite.com to mysite.com/catalog with a php file: <?php // redirect to catalog directory header("Location: http://mysite.com/catalog/index.php"); ?> <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html lang="en"> <head> <meta name="robots" content="noindex,follow"> </head> <body> </body> </html> Of course you have to change the http://mysite.com/catalog/index.php Link to comment Share on other sites More sharing options...
New 
Recommended Posts
Archived
This topic is now archived and is closed to further replies.