Jump to content
  • Checkout
  • Login
  • Get in touch


The e-commerce.

Pros/cons of using "robots.txt" to block spiders?


Recommended Posts

People seem to talk about "robots.txt" as if it's an absolute necessity, but I have yet to read a reason for writing one that has applied to my site. I'd love to be enlightened. :) I'm also wondering if there are any negative side effects to having a restrictive robots.txt file. (In terms of search engine optimization for example--is it better to disallow low-content pages like "login.php" or to have a completely "open" site? Or does it matter at all?)

Link to comment
Share on other sites

All I got to say is I used to have restrictions, took them off, and didn't see one bit of difference in server performance. Still running fast and furious. Maybe I'm just blessed with a good host, I don't know.

Go New England Patriots!!!!

Link to comment
Share on other sites


This topic is now archived and is closed to further replies.

  • Create New...