Jump to content
  • Checkout
  • Login
  • Get in touch

osCommerce

The e-commerce.

Spider Text


dario9

Recommended Posts

Hello I recently added new contributions for spiders into my "spider txt" but i have one question.

Is it really necessary to have these contributions in alphabetical order for the search engine spiders to crawl effectively??

 

I just copied and pasted my contributions as follows...

 

$Id: spiders.txt 1739 2007-12-20 00:52:16Z hpdl $

almaden.ibm.com

appie 1.1

architext

ask jeeves

asterias2.0

augurfind

baiduspider

bannana_bot

bdcindexer

crawler

crawler@fast

docomo

fast-webcrawler

fluffy the spider

frooglebot

geobot

googlebot

gulliver

henrythemiragorobot

ia_archiver

infoseek

kit_fireball

lachesis

lycos_spider

mantraagent

mercator

moget/1.0

muscatferret

nationaldirectory-webspider

naverrobot

ncsa beta

netresearchserver

ng/1.0

osis-project

polybot

pompos

scooter

seventwentyfour

sidewinder

sleek spider

slurp/si

[email protected]

steeler/1.3

szukacz

t-h-u-n-d-e-r-s-t-o-n-e

teoma

turnitinbot

ultraseek

vagabondo

voilabot

w3c_validator

zao/0

zyborg/1.0

booch

iltrovatore

linkalarm

nameprotect

poppelsdorf

scrubby

sohu

diamondbot

peerbot

searchbot

ebot

aspseek

StackRambler

smartwit

holmes

accoona

mj12

mediapartners

homerweb

lwp

ivia

/teoma

ichiro

volcano

wget

atlocal

objectssearch

multitext

shopwiki

pbot

boitho

dmoz

findlinks

falcon

ingrid

omni

sna-

sygol

mnogo

kbot

miva

mbot

tbot

bot.

cfetch

helix

sbider

ccubee

salty

spinner

ejupiter

heritrix

htdig

jakarta

blogslive

ping.blo.gs

topicblogs

poirot

silk

theophrastus

twiceler

updated

xirq

tarantula

wavefire

npbot

almaden.ibm.com

bdcindexer

become

Delorie.com

homer

kit_fireball

mnogo

poodle

worm

digger

mapoftheinternet

wire

csci

dtaagent

ebingbong

genieknows

najdi

zbot

noyona

sphider

ditto

Sensis Web Crawler

inktomisearch.com

comagent

psycheclone

yeti

walker

wwwster

dataparksearch

onetszukaj

page_verifier

python

adressendeutschland

kretrieve

openintelligencedata

biglotron

gralon

curl

minirank

yandex

combine

twisted

depspid

charlotte

publisher

microsoft url control

vbot

-bot

pagebull

pogodak

snappy

blaiz

yodao

nokia6682/

pansclient

vouager-hc

lbot

scoutjet

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Unfortunately, your content contains terms that we do not allow. Please edit your content to remove the highlighted words below.
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...