Jump to content
  • Checkout
  • Login
  • Get in touch

osCommerce

The e-commerce.

robots.txt help:(


helpme:(

Recommended Posts

Posted

I would just like to clarify if its even safe to have a robots.txt file as it basically outlines your whole site? If there is a topic on this or if someone could shed some light on this I would appreciate it.

 

Thanks :)

Posted

If you don't have a robots.txt, search engine spiders (and anyone else) can still follow each link and reference in your pages to see what's there. Keep in mind that robots.txt is only a suggestion for "well behaved" bots to keep out of certain areas -- it does not enforce no-fly zones. Two things to remember:

 

1) If some area of your site is not reachable from your pages (e.g., no link to "admin/"), it will never be seen by bots and should not be listed in robots.txt. If you want to reduce the chances of someone else seeing something, don't link to it, don't reference it, don't put it in robots.txt, and don't put it in a sitemap.

 

2) Something like "admin/" is well known to all the world, particularly once they know you're running osC, so unscrupulous bots will still go poking around there, robots.txt or no. You need to take other actions, such as password protecting areas or changing to unusual, unguessable names (both of which are recommended for "admin/").

 

One other thing: you can take measures to protect critical control files such as php.ini and perhaps .htaccess from being read by the outside world (generally done with commands in .htaccess).

 

Hackers will throw everything they can think of against a site, hoping that something sticks.

Posted

Phil,

 

Thanks for the advice, very informative. So, pages like "account.php" is not necessary in the txt file as there's no link to it on the site right? So in a fresh install of osC what pages should be in the txt file?

 

Appreciate the help. Thanks :)

Posted

Phil,

 

Thanks for the advice, very informative. So, pages like "account.php" is not necessary in the txt file as there's no link to it on the site right? So in a fresh install of osC what pages should be in the txt file?

 

Appreciate the help. Thanks :)

If there is no link to account.php, how do your previous customers log in?

 

There is an add on for robots text:

http://addons.oscommerce.com/info/2162

 

Also a neat IP trap for bad bots and snoops.

 

Tim

Posted

Tim,

 

Thanks for the link, we are actually have this txt file installed. So I'm guessing a fresh install of osC should contain these disallows? Is it ok to have these pages listed like that? The thing about the bots file is that it's out there for everyone to view... <_<

Posted

Tim,

 

Thanks for the link, we are actually have this txt file installed. So I'm guessing a fresh install of osC should contain these disallows? Is it ok to have these pages listed like that? The thing about the bots file is that it's out there for everyone to view... <_<

Read the reply from Mr Phil again.

 

My 2 cents is this: the robot file is for good bots only. Do not put any secret files in the text for bad bots or people to see. Everyone already knows you have an account.php and a reviews.php, so no secret there. When you change the name of your admin directory, do not put that in your text file.

 

Tim

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...