johnnybebad Posted January 9, 2010 Share Posted January 9, 2010 Okay, so I have my site/s working okay but I am getting a pounding from various so called search engines. So say i want to limit the search engines allowed to google, yahoo or bing or whatever the main three are these days. I have to let Mcafee scan my site completley to find security holes. I want to stop bots from accessing my images except for those that need to in order to promote my site, not sure say how shopmania/ google base get my images but as they are advertising my site I wouldnt want to stop thenm from having access. However I am not sure I want google images to have them, as thats not necessarily going to be used to find me. Can anyone advise the most efficient way of implimenting the above please. Thanks Getting better with mods but no programmer am I. Link to comment Share on other sites More sharing options...
♥mdtaylorlrim Posted January 9, 2010 Share Posted January 9, 2010 Okay, so I have my site/s working okay but I am getting a pounding from various so called search engines. So say i want to limit the search engines allowed to google, yahoo or bing or whatever the main three are these days. I have to let Mcafee scan my site completley to find security holes. I want to stop bots from accessing my images except for those that need to in order to promote my site, not sure say how shopmania/ google base get my images but as they are advertising my site I wouldnt want to stop thenm from having access. However I am not sure I want google images to have them, as thats not necessarily going to be used to find me. Can anyone advise the most efficient way of implimenting the above please. Thanks User-agent: * Disallow: / User-agent: bing Allow: / User-agent: Googlebot Allow: / User-agent: yahoo Allow: / Many will offer a lot of other rules with various pathways, restrictions to admin directory, etc. But, remember that the robots.txt file is available to hackers. It does no good to change the name of your admin directory if you advertise the name in a robots.txt file. BTW... these are not the true names of the yahoo or bing bots. You will have to find that out for yourself. Community Bootstrap Edition, Edge Avoid the most asked question. See How to Secure My Site and How do I...? Link to comment Share on other sites More sharing options...
Hotclutch Posted January 9, 2010 Share Posted January 9, 2010 Googlebot is correct, Bing is msnbot, Yahoo is slurp. If you don't want your images indexed by Google, then add: User-agent: Googlebot-Image Disallow: / Link to comment Share on other sites More sharing options...
johnnybebad Posted January 9, 2010 Author Share Posted January 9, 2010 Thanks i will look at that asap Getting better with mods but no programmer am I. Link to comment Share on other sites More sharing options...
johnnybebad Posted January 9, 2010 Author Share Posted January 9, 2010 Thanks i will look at that asap Getting better with mods but no programmer am I. Link to comment Share on other sites More sharing options...
johnnybebad Posted January 9, 2010 Author Share Posted January 9, 2010 found this link http://www.mcanerin.com/en/search-engine/robots-txt.asp to generate a robots.txt file, let us know if its any good and which of the bots would you allow or disallow? Getting better with mods but no programmer am I. Link to comment Share on other sites More sharing options...
♥mdtaylorlrim Posted January 10, 2010 Share Posted January 10, 2010 found this link http://www.mcanerin.com/en/search-engine/robots-txt.asp to generate a robots.txt file, let us know if its any good and which of the bots would you allow or disallow? You should probably use Google Webmaster Tools. I thought everybody did. Community Bootstrap Edition, Edge Avoid the most asked question. See How to Secure My Site and How do I...? Link to comment Share on other sites More sharing options...
New 
Recommended Posts
Archived
This topic is now archived and is closed to further replies.