Jump to content
  • Checkout
  • Login
  • Get in touch

osCommerce

The e-commerce.

Robots.txt ... I need a good template please!


phi148

Recommended Posts

  • 3 weeks later...
Would someone be kind enough to show me a really good robots.txt file... I'm really new at all this & really need some help :)

 

Thanks!

 

This is my one I don't know if this is any good I would be quite intrested in what other people think.

 

$Id: spiders.txt,v 1.2 2003/05/05 17:58:17 dgw_ Exp $

almaden.ibm.com

appie 1.1

architext

ask jeeves

asterias2.0

augurfind

baiduspider

bannana_bot

bdcindexer

crawler

crawler@fast

docomo

fast-webcrawler

fluffy the spider

frooglebot

geobot

googlebot

gulliver

henrythemiragorobot

ia_archiver

infoseek

kit_fireball

lachesis

lycos_spider

mantraagent

mercator

moget/1.0

muscatferret

nationaldirectory-webspider

naverrobot

ncsa beta

netresearchserver

ng/1.0

osis-project

polybot

pompos

scooter

seventwentyfour

sidewinder

sleek spider

slurp/si

[email protected]

steeler/1.3

szukacz

t-h-u-n-d-e-r-s-t-o-n-e

teoma

turnitinbot

ultraseek

vagabondo

voilabot

w3c_validator

zao/0

zyborg/1.0

crawl

slurp

spider

seek

ebot

obot

abot

dbot

hbot

kbot

mbot

nbot

pbot

rbot

sbot

tbot

ybot

zbot

bot.

bot/

_bot

.bot

accoona

adressendeutschland

appie

architext

asterias

atlocal

atomz

augurfind

bannana_bot

bdfetch

blo.

blog

boitho

booch

ccubee

Link to comment
Share on other sites

you posted your spiders.txt and not your robots.txt

 

User-agent: Googlebot-Image

Disallow: /

 

User-agent: *

Disallow: /account.php

Disallow: /advanced_search.php

Disallow: /checkout_shipping.php

Disallow: /cookie_usage.php

Disallow: /create_account.php

Disallow: /login.php

Disallow: /password_forgotten.php

Disallow: /popup_image.php

Disallow: /shopping_cart.php

Disallow: /images/

Disallow: /cgi-bin/

Disallow: /download/

Disallow: /product_reviews_write.php

 

and also in my admin directory, I have

 

User-agent: *

Disallow: /

Link to comment
Share on other sites

you posted your spiders.txt and not your robots.txt

 

User-agent: Googlebot-Image

Disallow: /

 

User-agent: *

Disallow: /account.php

Disallow: /advanced_search.php

Disallow: /checkout_shipping.php

Disallow: /cookie_usage.php

Disallow: /create_account.php

Disallow: /login.php

Disallow: /password_forgotten.php

Disallow: /popup_image.php

Disallow: /shopping_cart.php

Disallow: /images/

Disallow: /cgi-bin/

Disallow: /download/

Disallow: /product_reviews_write.php

 

and also in my admin directory, I have

 

User-agent: *

Disallow: /

 

:blush: :blush: :blush: :blush: :blush: :blush:

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...