Jump to content
  • Checkout
  • Login
  • Get in touch

osCommerce

The e-commerce.

Spiders / robots help


Guest

Recommended Posts

Hi everyone.

 

I'm starting to look at SEO for my site. I get fairly good page rankings already but I want to improve things.

 

I've seen robots / spiders mentioned. I realise that they are associated with Search engines, and that robots and spiders are how SEs catalogue your site.

 

I've read that its a good idea to have a robots file? What does this mean? What does the file do? Is it normally a result of an associated contribution, or is this a seperate step?

 

FWIW, I haven't done any SEO steps yet, and this is the planning stage for me.

 

(Do we have a robots/spiders FAQ?)

Link to comment
Share on other sites

Spiders.txt is basically just a list of search engines, check the contributions section for the lasted one. The robots.txt file just tells search engines where not to go, such as SSL pages, directories, etc.

Link to comment
Share on other sites

You can vistit any site that uses a robots file and see theirs by using http://somesite.com/robots.txt. There are sites around that explain how to create one too.

 

Jack

Support Links:

For Hire: Contact me for anything you need help with for your shop: upgrading, hosting, repairs, code written, etc.

All of My Addons

Get the latest versions of my addons

Recommended SEO Addons

Link to comment
Share on other sites

Thanks Jack. Yes I found this great resource: http://www.searchengineworld.com/robots/robots_tutorial.htm

 

So far I have this. I can't really think of any more.

 

User-agent: *

Disallow: /shop/admin

Disallow: /shop/adminbackups

Disallow: /shop/thumbnails

Disallow: /shop/tmp

 

The file works. Am I doing anything I shouldn't? I don't see a reason for the SEs to index /thumbnails or /tmp.

 

Is there anything I've missed?

Link to comment
Share on other sites

You could block your includes directory and all of the secured pages, like login and checkout since the SE's have no business going there. There is a recent thread which talks about how not to list your admin in the file since that gives the hackers a headstart. I don't have a link but if you do a search on robots it should show up.

 

Jack

Support Links:

For Hire: Contact me for anything you need help with for your shop: upgrading, hosting, repairs, code written, etc.

All of My Addons

Get the latest versions of my addons

Recommended SEO Addons

Link to comment
Share on other sites

Thanks for the tips guys.

 

OK I've included a seperate robots.txt in my "admin" folder.

 

This is my main robots.txt file. Am I doing anything wrong? I'm pretty sure its ok :)

 

All the actual .php pages that I've listed are secure ones, apart from products_reviews_write.php which I don't think the SEs need to see anyway.

 

User-agent: *

Disallow: /shop/adminbackups

Disallow: /shop/includes

Disallow: /shop/thumbnails

Disallow: /shop/tmp

Disallow: /shop/account.php

Disallow: /shop/account_edit.php

Disallow: /shop/account_history.php

Disallow: /shop/account_history_info.php

Disallow: /shop/account_newsletters.php

Disallow: /shop/account_notifications.php

Disallow: /shop/account_password.php

Disallow: /shop/address_book.php

Disallow: /shop/address_book_process.php

Disallow: /shop/checkout.php

Disallow: /shop/checkout_shipping.php

Disallow: /shop/create_account.php

Disallow: /shop/login.php

Disallow: /shop/product_reviews_write.php

Disallow: /shop/shopping_cart.php

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...