Jump to content
  • Checkout
  • Login
  • Get in touch


The e-commerce.

How is this possible (spider,crawler)


Recommended Posts

Hello all,


Could someone explain me how this works?


Today I noticed a (spider, robot) that was filling the users shopping cart in the who?s online.


And suddenly when I was looking at some of my products I saw that the shopping cart was filled with products. That is not good if a custommer sees products in his cart that he has not put in there by himself.


I have the prevent spider session set to true.


Somebody explain this to me please.


Thanyou all,

Link to comment
Share on other sites

Hi everyone, thanks for the tips on robots.txt and spiders.txt. But what if I WANT google and MSN to spider my site but don't want it to add more than 50 cart items? Is there a way to do that?


My ISP has suspended my account because of these bots. I'm guessing that it's because the bot has 50 items on it's cart and everytime it goes to the next page, ALL 50 items have to be read from the database. And these just gets bigger and bigger.


I figure if I do this, then it won't be so taxing on the database.


Does anyone have a contrib or quick hack to limit the number of products in aa shopping cart to 50? Assuming of course that regular shoppers would never have 50 products on their cart.

Link to comment
Share on other sites



Install the spiders.txt referenced in yesudo's post above.

Go into admin->Configuration->Sessions and set Prevent Spider Sessions to true

Make sure that robots.txt (under /catalog) includes shopping_cart.php


The robots.txt file should stop the bots from clicking the Add To Cart button.

The Prevent Spider Sessions=true and the spiders.txt file will stop bots from seeing the osCid param in the URL. Everytime they come in and get a new osCid, they think it is a new page so they re-index your site over and over again. They also store the whole URL in their index which isn't good.


The 2 of these things should lower the bot traffic on your site. It will also allow the bots to crawl your site without loading up a cart.


Good luck,


Link to comment
Share on other sites


This topic is now archived and is closed to further replies.

  • Create New...