dario9 Posted May 12, 2008 Posted May 12, 2008 Hello I recently added new contributions for spiders into my "spider txt" but i have one question. Is it really necessary to have these contributions in alphabetical order for the search engine spiders to crawl effectively?? I just copied and pasted my contributions as follows... $Id: spiders.txt 1739 2007-12-20 00:52:16Z hpdl $ almaden.ibm.com appie 1.1 architext ask jeeves asterias2.0 augurfind baiduspider bannana_bot bdcindexer crawler crawler@fast docomo fast-webcrawler fluffy the spider frooglebot geobot googlebot gulliver henrythemiragorobot ia_archiver infoseek kit_fireball lachesis lycos_spider mantraagent mercator moget/1.0 muscatferret nationaldirectory-webspider naverrobot ncsa beta netresearchserver ng/1.0 osis-project polybot pompos scooter seventwentyfour sidewinder sleek spider slurp/si [email protected] steeler/1.3 szukacz t-h-u-n-d-e-r-s-t-o-n-e teoma turnitinbot ultraseek vagabondo voilabot w3c_validator zao/0 zyborg/1.0 booch iltrovatore linkalarm nameprotect poppelsdorf scrubby sohu diamondbot peerbot searchbot ebot aspseek StackRambler smartwit holmes accoona mj12 mediapartners homerweb lwp ivia /teoma ichiro volcano wget atlocal objectssearch multitext shopwiki pbot boitho dmoz findlinks falcon ingrid omni sna- sygol mnogo kbot miva mbot tbot bot. cfetch helix sbider ccubee salty spinner ejupiter heritrix htdig jakarta blogslive ping.blo.gs topicblogs poirot silk theophrastus twiceler updated xirq tarantula wavefire npbot almaden.ibm.com bdcindexer become Delorie.com homer kit_fireball mnogo poodle worm digger mapoftheinternet wire csci dtaagent ebingbong genieknows najdi zbot noyona sphider ditto Sensis Web Crawler inktomisearch.com comagent psycheclone yeti walker wwwster dataparksearch onetszukaj page_verifier python adressendeutschland kretrieve openintelligencedata biglotron gralon curl minirank yandex combine twisted depspid charlotte publisher microsoft url control vbot -bot pagebull pogodak snappy blaiz yodao nokia6682/ pansclient vouager-hc lbot scoutjet Quote
New 
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.