Jump to content
  • Checkout
  • Login
  • Get in touch

osCommerce

The e-commerce.

Search Engines - Am I being indexed?


kghammond

Recommended Posts

I added a robots.txt file that basically told the search engines to not index the admin directory. Otherwise it should parse the whole catalog.

 

My main page:

http:\\www.matrixtek.com has two links, one to the catalog and one to the allprods.php page.

 

I submitted the site to all the major search engines. I am getting visits from the bots, but I am not sure if they are working. Only one or two seem to be asking for the actual product pages or the all product page.

 

Any advice would be appeciated.

 

Thank You,

Kevin

 

Here are some of the visits from the logs:

207.31.251.140 - - [02/Apr/2005:10:18:24 -0500] "GET /robots.txt HTTP/1.1" 404 216 "-" "NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot)"

207.31.251.140 - - [02/Apr/2005:10:18:34 -0500] "GET / HTTP/1.1" 403 210 "-" "NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot)"

 

www.whois.sc - - [04/Apr/2005:04:03:18 -0400] "GET /robots.txt HTTP/1.1" 404 216 "http://www.whois.sc/" "SurveyBot/2.3 (Whois Source)"

www.whois.sc - - [04/Apr/2005:04:03:18 -0400] "GET / HTTP/1.1" 206 2300 "http://www.whois.sc/matrixtek.com" "SurveyBot/2.3 (Whois Source)"

 

scrubby.scrubtheweb.com - - [05/Apr/2005:00:27:48 -0400] "GET /robots.txt HTTP/1.1" 404 216 "-" "Scrubby/2.1 (http://www.scrubtheweb.com/abs/meta-check.html)"

scrubby1.scrubtheweb.com - - [05/Apr/2005:00:27:48 -0400] "GET / HTTP/1.1" 206 2411 "-" "Scrubby/2.1 (http://www.scrubtheweb.com/abs/meta-check.html)"

 

msnbot.msn.com - - [05/Apr/2005:02:21:08 -0400] "GET / HTTP/1.0" 200 2411 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)"

msnbot.msn.com - - [05/Apr/2005:02:21:08 -0400] "GET /robots.txt HTTP/1.0" 200 220 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)"

 

lj2547.inktomisearch.com - - [06/Apr/2005:02:40:56 -0400] "GET /robots.txt HTTP/1.0" 200 220 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"

lj2113.inktomisearch.com - - [06/Apr/2005:02:40:58 -0400] "GET /osCommerce/allprods.php HTTP/1.0" 200 18489 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"

 

lj1124.inktomisearch.com - - [08/Apr/2005:20:09:29 -0400] "GET /robots.txt HTTP/1.0" 200 220 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"

 

crawl26-public.alexa.com - - [08/Apr/2005:20:21:20 -0400] "GET /robots.txt HTTP/1.0" 200 220 "-" "ia_archiver"

crawl26-public.alexa.com - - [08/Apr/2005:20:21:20 -0400] "GET / HTTP/1.0" 200 2411 "-" "ia_archiver"

crawl26-public.alexa.com - - [08/Apr/2005:21:20:39 -0400] "GET /robots.txt HTTP/1.0" 200 220 "-" "ia_archiver"

crawl26-public.alexa.com - - [08/Apr/2005:21:20:39 -0400] "GET /osCommerce HTTP/1.0" 301 240 "-" "ia_archiver"

 

crawl-66-249-64-28.googlebot.com - - [09/Apr/2005:10:28:16 -0400] "GET /robots.txt HTTP/1.0" 200 220 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"

crawl-66-249-64-28.googlebot.com - - [09/Apr/2005:10:28:17 -0400] "GET / HTTP/1.0" 200 2411 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"

crawl-66-249-71-40.googlebot.com - - [09/Apr/2005:12:02:56 -0400] "GET /osCommerce HTTP/1.0" 301 240 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"

crawl-66-249-71-69.googlebot.com - - [09/Apr/2005:13:52:27 -0400] "GET /osCommerce/allprods.php HTTP/1.0" 200 18488 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"

crawl-66-249-64-30.googlebot.com - - [09/Apr/2005:14:27:51 -0400] "GET /osCommerce/ HTTP/1.0" 200 19701 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"

Link to comment
Share on other sites

you probably dont want the indexing the images, download, pub, CVS, login.php, shopping_cart.php, whole bunch of them. there is a robots.txt file in the contributions

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...