Tubbing Posted May 9, 2008 Share Posted May 9, 2008 Hi, I just want to make a test page, and set a new css for it to play around with looks and such, but dont want any bots wasting their time crawling it or search engines storing it. How would I go about it? Or if I just make a new page and not place a link to it in any info box will it not be crawled then??? Does google bot pick up on hiden pageS? THanks for any help. Regards Tubz Link to comment Share on other sites More sharing options...
Guest Posted May 9, 2008 Share Posted May 9, 2008 Create a file called robots.txt In that file put the following in it User-agent: * Disallow: / That will tell the bots not to index your site. However not all robots pay attention to that. Don't forget to make changes to that file once you are ready with your site. For mor info on robots.txt look at this site http://www.robotstxt.org/robotstxt.html Link to comment Share on other sites More sharing options...
germ Posted May 9, 2008 Share Posted May 9, 2008 If this page isn't linked to in any of your other pages you don't have to worry - they won't find it. You have to understand how bots work. All they do is go to http://yourdomain.com/ Then, whatever page comes up (index.html or index.php) they just start following links, unless you have the meta-tag telling them not to, or the page is disallowed in the robots.txt file. At least the "good bots" follow robots.txt They can't see all the pages you have, and they don't guess. They just follow links. So if this page isn't linked to by anything the robot does index, you don't have to disallow it because they'll never find it. I have a lot of "test" pages for one thing or another and they never get crawled because they aren't linked to by anything the robots do crawl. ;) If I suggest you edit any file(s) make a backup first - I'm not perfect and neither are you. "Given enough impetus a parallelogramatically shaped projectile can egress a circular orifice." - Me - "Headers already sent" - The definitive help "Cannot redeclare ..." - How to find/fix it SSL Implementation Help Like this post? "Like" it again over there > Link to comment Share on other sites More sharing options...
Guest Posted May 9, 2008 Share Posted May 9, 2008 Thanks for that info Jim. I was not aware of that. Very good to know though. Link to comment Share on other sites More sharing options...
Tubbing Posted May 9, 2008 Author Share Posted May 9, 2008 Thanks jim, thats perfect... I can start testing away... Do bots follow the calls to Css files then??? or is that a dumb question, probly is coz ive never seen a css file in google :). Link to comment Share on other sites More sharing options...
germ Posted May 9, 2008 Share Posted May 9, 2008 You can use the same CSS files used by your osC shop and they still won't get crawled because the CSS file doesn't "link back". Bots are just "automated link followers". Nothing more - nothing less. :lol: If I suggest you edit any file(s) make a backup first - I'm not perfect and neither are you. "Given enough impetus a parallelogramatically shaped projectile can egress a circular orifice." - Me - "Headers already sent" - The definitive help "Cannot redeclare ..." - How to find/fix it SSL Implementation Help Like this post? "Like" it again over there > Link to comment Share on other sites More sharing options...
Qim Posted July 18, 2008 Share Posted July 18, 2008 You can use the same CSS files used by your osC shop and they still won't get crawled because the CSS file doesn't "link back". Bots are just "automated link followers". Nothing more - nothing less. :lol: Hi, good to know! Thanks... :) For those who have like www.myshop.com/shop and do redirect so that www.myshop.com will point to the above from .htaccess will that also prevent spiders? Regards Sara Link to comment Share on other sites More sharing options...
germ Posted July 19, 2008 Share Posted July 19, 2008 If there is no link in your store to it, it won't get spidered. I'm not sure I understand exactly what it is you're talking about preventing getting spidered. :unsure: If I suggest you edit any file(s) make a backup first - I'm not perfect and neither are you. "Given enough impetus a parallelogramatically shaped projectile can egress a circular orifice." - Me - "Headers already sent" - The definitive help "Cannot redeclare ..." - How to find/fix it SSL Implementation Help Like this post? "Like" it again over there > Link to comment Share on other sites More sharing options...
New 
Recommended Posts
Archived
This topic is now archived and is closed to further replies.