Jump to content
  • Checkout
  • Login
  • Get in touch


The e-commerce.

Quick spider question.


Recommended Posts

Theoretically they should visit those pages that have the "robots" tag set to "follow".


Then again some :devil: use custom spiders they do not follow tags while scanning domains and of course its impossible to detect.

Link to comment
Share on other sites


This topic is now archived and is closed to further replies.

  • Create New...