lee246 Posted August 18, 2005 Share Posted August 18, 2005 Is there anyway I can see what pages the spiders are visiting on my website? Thanks. Link to comment Share on other sites More sharing options...
Guest Posted August 18, 2005 Share Posted August 18, 2005 Theoretically they should visit those pages that have the "robots" tag set to "follow". Then again some :devil: use custom spiders they do not follow tags while scanning domains and of course its impossible to detect. Link to comment Share on other sites More sharing options...
New 
Recommended Posts
Archived
This topic is now archived and is closed to further replies.