phi148 Posted October 3, 2006 Posted October 3, 2006 Would someone be kind enough to show me a really good robots.txt file... I'm really new at all this & really need some help :) Thanks!
JemCraft Posted October 18, 2006 Posted October 18, 2006 Would someone be kind enough to show me a really good robots.txt file... I'm really new at all this & really need some help :) Thanks! I havent tried it myself yet, but I found this contribution http://www.oscommerce.com/community/contributions,2162 Jemma
Chris Smith Posted October 18, 2006 Posted October 18, 2006 Would someone be kind enough to show me a really good robots.txt file... I'm really new at all this & really need some help :) Thanks! This is my one I don't know if this is any good I would be quite intrested in what other people think. $Id: spiders.txt,v 1.2 2003/05/05 17:58:17 dgw_ Exp $ almaden.ibm.com appie 1.1 architext ask jeeves asterias2.0 augurfind baiduspider bannana_bot bdcindexer crawler crawler@fast docomo fast-webcrawler fluffy the spider frooglebot geobot googlebot gulliver henrythemiragorobot ia_archiver infoseek kit_fireball lachesis lycos_spider mantraagent mercator moget/1.0 muscatferret nationaldirectory-webspider naverrobot ncsa beta netresearchserver ng/1.0 osis-project polybot pompos scooter seventwentyfour sidewinder sleek spider slurp/si [email protected] steeler/1.3 szukacz t-h-u-n-d-e-r-s-t-o-n-e teoma turnitinbot ultraseek vagabondo voilabot w3c_validator zao/0 zyborg/1.0 crawl slurp spider seek ebot obot abot dbot hbot kbot mbot nbot pbot rbot sbot tbot ybot zbot bot. bot/ _bot .bot accoona adressendeutschland appie architext asterias atlocal atomz augurfind bannana_bot bdfetch blo. blog boitho booch ccubee
Guest Posted October 18, 2006 Posted October 18, 2006 you posted your spiders.txt and not your robots.txt User-agent: Googlebot-Image Disallow: / User-agent: * Disallow: /account.php Disallow: /advanced_search.php Disallow: /checkout_shipping.php Disallow: /cookie_usage.php Disallow: /create_account.php Disallow: /login.php Disallow: /password_forgotten.php Disallow: /popup_image.php Disallow: /shopping_cart.php Disallow: /images/ Disallow: /cgi-bin/ Disallow: /download/ Disallow: /product_reviews_write.php and also in my admin directory, I have User-agent: * Disallow: /
Chris Smith Posted October 18, 2006 Posted October 18, 2006 you posted your spiders.txt and not your robots.txt User-agent: Googlebot-Image Disallow: / User-agent: * Disallow: /account.php Disallow: /advanced_search.php Disallow: /checkout_shipping.php Disallow: /cookie_usage.php Disallow: /create_account.php Disallow: /login.php Disallow: /password_forgotten.php Disallow: /popup_image.php Disallow: /shopping_cart.php Disallow: /images/ Disallow: /cgi-bin/ Disallow: /download/ Disallow: /product_reviews_write.php and also in my admin directory, I have User-agent: * Disallow: / :blush: :blush: :blush: :blush: :blush: :blush:
New 
Recommended Posts
Archived
This topic is now archived and is closed to further replies.