Spipers keep getting cookie usage page

[damo2576]

Hi,

 

How can I stop this? Any ideas?

 

All help/suggesions appreciated!

 

Damo

[Guest]

Set Allow spider sessions to FALSE in configuration

[Guest]

Sorry brain fart ,set prevent spider sessions to true

[damo2576]

Hi,

 

Thanks for reply - prevent spider session is set to true already. None of the pages get a session id, i also have set cookies set to true.

 

I think its when the spider follows a link to buy or something like that and since it cant be able to set a cookie it comes up against the cookie usage page.

 

any ideas?

 

Damo

[damo2576]

Incidentally I've just been searching for help on this on google and have come across literally 100's of people using OSC's cookie usage pages.

 

There must be a way around this problem?

 

Damo

[damo2576]

sorry to bump this but does anyone have a solution? i don't really want my cookie usage page in google and sure others dont too?

 

cheers

 

damo

[dr_lucas]

I am having exactly the same problem. Looking in my User Tracking cotrtibution I see googlebot is accessing the cookie usage page at least 100 times in 1 hour, not trying to move to other pages, and then randomally move to index other pages and return back to cookie usage. Here is an example:

 

Click Count:  1081 

Start Time:  23:59:33  Idle Time:  00:00:20

End Time:  20:49:55  Total Time:  20:50:22

Country:  United States United States

IP Address:  64.68.83.153

Host:  crawl34.googlebot.com

 

 

# 16:15:20:  Closeouts

  /index.php?cPath=27&sort=3d&page=1

16:13:36:  Cookie Usage  /cookie_usage.php

16:13:06:  Cookie Usage  /cookie_usage.php

16:12:37:  ZB New Collection /product_info.php?products_id=146

16:12:03:  Cookie Usage  /cookie_usage.php

16:11:33:  Cookie Usage  /cookie_usage.php

16:11:10:  Cookie Usage  /cookie_usage.php

16:10:49:  Cookie Usage  /cookie_usage.php

16:10:25:  Cookie Usage  /cookie_usage.php

16:09:58:  Cookie Usage  /cookie_usage.php

16:09:25:  Cookie Usage  /cookie_usage.php

16:08:57:  Cookie Usage  /cookie_usage.php

16:08:29:  Cookie Usage  /cookie_usage.php

16:07:56:  Cookie Usage  /cookie_usage.php

16:07:26:  Cookie Usage  /cookie_usage.php

16:07:00:  Cookie Usage  /cookie_usage.php

16:06:29:  Cookie Usage  /cookie_usage.php

16:06:10:  Cookie Usage  /cookie_usage.php

16:05:49:  Cookie Usage  /cookie_usage.php

16:05:26:  Cookie Usage  /cookie_usage.php

16:05:00:  Cookie Usage  /cookie_usage.php

16:04:39:  Cookie Usage  /cookie_usage.php

16:04:10:  Cookie Usage  /cookie_usage.php

16:03:38:  Cookie Usage  /cookie_usage.php

16:03:03:  Cookie Usage  /cookie_usage.php

16:02:44:  Cookie Usage  /cookie_usage.php

16:02:24:  Cookie Usage  /cookie_usage.php

16:02:07:  Cookie Usage  /cookie_usage.php

16:01:54:  Cookie Usage  /cookie_usage.php

16:01:38:  Cookie Usage  /cookie_usage.php

16:01:09:  Cookie Usage  /cookie_usage.php

16:00:42:  Cookie Usage  /cookie_usage.php

16:00:13:  Cookie Usage  /cookie_usage.php

15:59:38:  Cookie Usage  /cookie_usage.php

15:59:08:  Cookie Usage  /cookie_usage.php

15:58:38:  Cookie Usage  /cookie_usage.php

15:58:21:  Cookie Usage  /cookie_usage.php

15:58:00:  Cookie Usage  /cookie_usage.php

15:57:36:  Cookie Usage  /cookie_usage.php

15:56:40:  Cookie Usage  /cookie_usage.php

15:56:13:  Cookie Usage  /cookie_usage.php

15:55:43:  Cookie Usage  /cookie_usage.php

15:54:42:  Cookie Usage  /cookie_usage.php

15:54:00:  Cookie Usage  /cookie_usage.php

15:53:26:  Cookie Usage  /cookie_usage.php

15:52:59:  Cookie Usage  /cookie_usage.php

15:52:26:  Cookie Usage  /cookie_usage.php

15:52:06:  Cookie Usage  /cookie_usage.php

15:51:42:  Cookie Usage  /cookie_usage.php

15:51:30:  Cookie Usage  /cookie_usage.php

15:51:05:  Cookie Usage  /cookie_usage.php

15:50:41:  Cookie Usage  /cookie_usage.php

15:50:22:  Cookie Usage  /cookie_usage.php

15:49:52:  Cookie Usage  /cookie_usage.php

15:49:00:  Cookie Usage  /cookie_usage.php

15:48:34:  Cookie Usage  /cookie_usage.php

15:48:09:  Cookie Usage  /cookie_usage.php

15:47:43:  Cookie Usage  /cookie_usage.php

15:47:24:  Cookie Usage  /cookie_usage.php

15:46:57:  Cookie Usage  /cookie_usage.php

15:46:30:  Cookie Usage  /cookie_usage.php

15:46:10:  Cookie Usage  /cookie_usage.php

15:45:48:  Cookie Usage  /cookie_usage.php

15:45:25:  Cookie Usage  /cookie_usage.php

15:45:02:  Cookie Usage  /cookie_usage.php

15:44:38:  Cookie Usage  /cookie_usage.php

15:44:17:  Cookie Usage  /cookie_usage.php

15:44:00:  Cookie Usage  /cookie_usage.php

15:43:46:  Cookie Usage  /cookie_usage.php

15:43:26:  Cookie Usage  /cookie_usage.php

15:43:02:  Cookie Usage  /cookie_usage.php

15:42:39:  Cookie Usage  /cookie_usage.php

15:42:18:  Cookie Usage  /cookie_usage.php

15:42:01:  Cookie Usage  /cookie_usage.php

15:41:21:  Cookie Usage  /cookie_usage.php

15:41:01:  Cookie Usage  /cookie_usage.php

15:40:38:  Cookie Usage  /cookie_usage.php

15:39:58:  Cookie Usage  /cookie_usage.php

15:39:38:  Cookie Usage  /cookie_usage.php

15:39:15:  Cookie Usage  /cookie_usage.php

15:38:52:  Cookie Usage  /cookie_usage.php

15:38:26:  Cookie Usage  /cookie_usage.php

15:38:03:  Trendy Studs            /product_info.php?products_id=72

 

Any ideas?

[mark27uk3]

Why not just add this to the robots.txt file

 

Disallow: /cookie_usage.php

 

Mark

[niknakgroup]

Or, be really cheeky and use it as part of your SEO campaign - I have the same problem and Google is by far the worst offendor (I currently have 6 bloddy googlebots in my store and it is killing my server, but what do you do....ban them, have no traffic, allow them, have slow site......??) but I have configured my cookie_usage.php page to be crammed with all of my keywords and target phrases. As I know the spiders will visit it regularly, it will have a high ranking in their listings for pages within my site, so why not use it......Just an idea! :D

[mark27uk3]

Please post your cookie_usage.php so others can see what you mean.

 

Mark

[niknakgroup]

includes/languages/english/cookie_usage.php:

<?php
/*
 $Id: cookie_usage.php,v 1.1 2003/03/10 23:32:20 hpdl Exp $

 osCommerce, Open Source E-Commerce Solutions
 http://www.oscommerce.com

 Copyright (c) 2003 osCommerce

 Released under the GNU General Public License
*/

define('NAVBAR_TITLE', 'Cookie Usage');
define('HEADING_TITLE', 'Cookie Usage');

define('TEXT_INFORMATION', 'We have detected that your browser does not support cookies, or has set cookies to be disabled.<br><br>To continue shopping online, we encourage you to enable cookies on your browser.<br><br>For <b>Internet Explorer</b> browsers, please follow these instructions:<br><ol><li>Click on the Tools menubar, and select Internet Options</li><li>Select the Security tab, and reset the security level to Medium</li></ol>We have taken this measurement of security for your benefit, and apologize upfront if any inconveniences are caused.<br><br>Please contact the store owner if you have any questions relating to this requirement, or to continue purchasing products offline.<p><b>JEWELLERY</b><p>
niknakgroup.com are one of the UKs leading online jewellery and gifts retailers. The jewellery range is very varied and caters for the jewellery and gift needs of all fashion conscious people.
<p> 
<a href="jewellery/silver"><b>Sterling Silver Jewellery</a>
</b><br>
Jewellery, jewelry, jewellry, jewelery....no matter how you spell it, weve got loads of it!! With a collection of over 5000 silver jewellery items you should be spoilt for choice, whether its silver jewellery for yourself or silver jewellery as a gift for someone special. The jewellery range includes a wide array of earrings, pendants, bracelets, rings and lots more in wide and varied themes. The styles vary from highly contemporary and fashionable  silver jewellery to simple silver jewellery designs and so caters for all fashion conscious women and men alike.
<p>
<a href="jewellery/gold"><b>Gold Jewellery</a>
</b><br>
An ever expanding gold jewellery range in both 9 and 18 carat gold. Our gold jewellery is a very high standard of design and good value for money. The niknakgroup.com gold jewellery range now boasts over 2500 jewellery items and with new jewellery additions arriving constantly, our gold jewellery range easily rivals anything else seen on the Web.
<p>
<a href="jewellery/Body+Jewellery"><b>Body Jewellery / Belly Bars</a>
</b><br>
niknakgroup.com now offers a very wide and extensive collection of body jewellery and belly bars. The body jewellery ranges now include: steel body jewellery, titanium body jewellery, 9ct gold body jewellery, 14ct gold body jewellery, 18ct gold body jewellery, ultra violet body jewellery, and sterling silver body jewellery. From navel bars to flesh plugs, the body jewellery range caters for all your body jewellery and body piercing needs. All of our body jewellery is sourced from the worlds largest manufacture of body jewellery, and as with all of our items is covered by a comprehensive guarantee and excellent product knowledge. We have over 1000 designs in countless variations and are absolutely certain that not only dow e carry one of the largest product ranges available anywhere, but that we also have extremely competetive pricing and have prices you will find extremely difficult to beat - either on the high street or on the internet.
<p>
<b>Other Designer Jewellery - Designer Jewellery
</b><br>
New to the website is a stunning collection of handmade contemporary designer jewellery. Specialising in sterling silver mixed with 9ct gold - it provides a truly breath-taking collection of designer jewellery. Mixing some of these styles with diamonds, semi-precious stones and even stones such as slate or granite really adds to the uniqueness of this special jewellery collection.
<p>
<b>Silver & Diamonds Jewellery - Diamond Jewellery
</b><br>
Our silver and diamonds range keeps getting bigger!! The full collection includes our Diamonds with Love ranges, D for Diamonds ranges just for children, the Kit Heath White Diamonds ranges (adults and children) and a great diamond range for weddings and bridesmaids. They all include spectacular jewellery designs in a multitude of silver finishes and stone settings to complement the quality diamonds. all the ranges are supplied in excellent gift boxes to make the diamond jewellery just a little bit more special.
<p>
<b>Kit Heath Jewellery
</b><br>
niknakgroup is one of a very few places you will find the ENTIRE Kit Heath collection. The Kit Heath jewellery range is a vibrant collection of contemporary jewellery in a variety of styles and finishes, and one of the most serached for designers of jewellery on the internet worldwide. There are the diamond collections, kids jewellery and of course their contemporary jewellery ranges, all of which can be found on this website.
<p>
<b>Mens Jewellery
</b><br>
The mens jewellery collections have gradually grown and now boasts a range of over 300 silver jewellery items just for men. From chunky bracelets to cufflinks to the very popular dog-tag pendants, the jewellery range caters for every man?s silver jewellery needs.
<p>
<b>Childrens Jewellery - Kids Jewellery - Baby Jewellery
</b><br>
Due to the over-whelming success of of childrens jewellery, we have now added lots more kids jewellery items including, earrings, studs, pendants, charm pendants, charm bracelets and lots more. There is also diamond jewellery collections for children from the D for Diamond range and by Kit Heath also.
<p>
<b>Stainless Steel & Titanium Rings - Urban Jewellery
</b><br>
There has been a recent trend towards steel and titanium jewellery, and niknakgroup.com has now intoduced a stunning stainless steel jewellery collection inspired by Kit Heath Jewellery. There are trendy design steel rings, steel earrings and steel pendants. The titanium collection consists of a range of titanium rings............but what makes our range so different is that though we do sell plain titanium bands, we also sell titanium rings inset with silver, or gold, or diamonds or a combination of them all. The jewellery collection here is a must see!
<p>
<b>Wedding Day Jewellery Range - Bridesmaids Jewellery
</b><br>
Every year we have been asked for more wedding day jewellery, more bridesmaids jewellery, more jewellery gifts. So now we have introduced a whole collection dedicated just to weddings. The jewellery range includes earrings, rings, pendants, necklaces, tiaras and bracelets. All set in a variety of silver finishes, variety of semi-precious stone settings, diamond jewellery and lots more. Elegant and neutral in colouring, this range of exquisite jewellery is sure to be a best seller, and will complement every wedding colour scheme with delicate pearls and diamonds set in sterling silver and gold. The designers of our wedding jewellery collection have kept in mind that whilst you want your wedding day gems to be beautiful and stunning, they should not outshine the beauty of the bride and bridesmaids! We have also added a great collection of wedding day jewellery for the Groom and Best Man.
<p>
<b>Pewter Jewellery
</b><br>
niknakgroup.com has now expanded into the Pewter jewellery range, and is proud to announce our collection of over 500 high quality jewellery designs in pewter with styles from Gothic and Wicca, including pentagrams, cannabis leafs and other pagan symbolism, through to skater wear and highly contemporary well priced pewter necklaces. Our collection of pewter jewellery contains Celtic and Mystic pieces along with the jewellery designs for todays more youthful customers.
<p>
<b>Celtic Jewellery
</b><br>
Having highlighted the Celtic designs as a firm favourite of our customers, niknakgroup.com have actively searched for quality suppliers who can add more designs to our extensive Celtic Jewellery range. The ancient artforms and symbolism of the Celts has been diligently replicated in our designers pieces, and we are absolutely certain that you will find your perfect Celtic piece right here. Available in gold, silver and pewter, our celtic designs feature diamonds, amethyst, topaz, cubic zirconia and many more gems and our celtic rings have been highlighted as one of the best collections on the internet.
<p>
<b>Rennie Mackintosh Jewellery
</b><br>
This architect from the turn of the 20th Century dedicated his professional life to functional and elegant design. Whilst Charles Rennie Mackintosh himself concentrated on buildings and furniture, the stylings of his work have been faithfully translated into exceptional jewellery designs. Our Rennie Mackintosh jewellery range brings the very best styles from many British designers who have all tried with great success to put Rennie Mackintoshs stylistic designs into high quality jewellery pieces in sterling silver.</font>');

define('BOX_INFORMATION_HEADING', 'Cookie Privacy and Security');
define('BOX_INFORMATION', 'Cookies must be enabled to purchase online on this store to embrace privacy and security related issues regarding your visit to this site.<br><br>By enabling cookie support on your browser, the communication between you and this site is strengthened to be certain it is you who are making transactions on your own behalf, and to prevent leakage of your privacy information.');
?>

 

see it at www.niknakgroup.com/jewellery/cookie_usage.php - the customer would still see the information they need, but the spider has lost of new links and text to chew on! :blink:

[John-Doe]

Looking in my User Tracking cotrtibution I see googlebot is accessing the cookie usage page at least 100 times in 1 hour, not trying to move to other pages

Not sure, but I think the "buy now" links to forms v1.2 contrib would help.

 

http://www.oscommerce.com/community/contributions,952/

[xpetex]

does this only apply if you have cookies enabled?

[niknakgroup]

I have had cookies turned off in my site since Christmas, but they still keep getting the page; I have tried replicating what they are doing myself, and cant get the page to show up no matter what I do.......oh well, just another of those little mysteries..... <_<

[John-Doe]

As far as I know it only happens if the sessions and cookies both don't work. And that is the case for the spiders (if you've enabled "Prevent Spider Sessions" and the spider is in your spiderlist).

 

And it does not happen to you (or your customers) because you are not in the spiderlist like the spiders, so for you the sessions work as they should.

[dr_lucas]

Why not just add this to the robots.txt file

 

Disallow: /cookie_usage.php

 

Mark

I don't have any robots.txt file, but I do have spiders.txt in my /catalog/includes

Is that the same?

[dr_lucas]

does this only apply if you have cookies enabled?

I have "Prevent Spider Sessions" set to True, and Force Cookie Use set to False.

[mark27uk3]

Have a look at this contribution

 

Mark

[dr_lucas]

Thanks a lot, Mark! :)

[mark27uk3]

No Problem Mikey!

 

If you need anymore help just ask!

 

Mark

[jon_l]

What I don't understand, is why a spider is being directed to the cookie_usage page. It looks like this is happening in application_top, which redirects if there isn't an active session.

 

Surely it should not redirect if its a spider?

 

Jon.

[jon_l]

OK, now I do understand why this is happening.

 

If a spider comes along and its not in the spiders.txt list, it will end up being directed to cookie_usage.php.

 

So, the simple solution is to look through your weblogs and identify the user agent on all requests for cookie_usage.php.

 

You should find that the majority, if not all, are spiders / robots.

 

Then simply add pattern matching for them to your spiders.txt file.

 

Jon.

[gert cuykens]

where do i find my web logs on a ipower hosted server ?

[jon_l]

Ignore my posts above, I was talking cr*p.

 

After checking my weblogs, all of the cookie_usage hits were from Googlebot and as a result of it trying to follow the buy_it_now link. I'm going to try the fix noted earlier on in this thread, and will report back with the results.

 

If you don't know where to find your weblogs, contact your host. Or try the support section of their website, it should tell you where to find them (if they are available). Though if you have ftp access, you might be able to find them on your own.

 

Jon.

[ari]

Hi guys,

 

I tried this on another thread, perhaps someone here has an answer. My system does not create a session for googlebot, yet the bot keeps trying to "buy" products and then it hits the cookie usage page.

 

Here's some sample log entries (some are legitimate traffic)

 

66.249.79.63 - - [17/Oct/2004:02:01:39 -0700] "GET /index.php?cPath=1_27&page=1&sort=5a&action=buy_now&products_id=258 HTTP/1.0" 302 0 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

66.249.79.63 - - [17/Oct/2004:02:01:39 -0700] "GET /cookie_usage.php HTTP/1.0" 200 21764 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

66.249.78.59 - - [17/Oct/2004:02:01:54 -0700] "GET /index.php?cPath=1_27&page=1&sort=5d HTTP/1.0" 200 28434 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

66.207.118.206 - - [17/Oct/2004:02:02:05 -0700] "GET /product_info.php?products_id=505&osCsid=472a70b19e8107dab70be0b6c3ee42e2 HTTP/1.1" 200 27914 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)"

66.249.78.59 - - [17/Oct/2004:02:02:27 -0700] "GET /index.php?cPath=1_27&page=1&sort=5a&action=buy_now&products_id=138 HTTP/1.0" 302 0 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

66.249.78.59 - - [17/Oct/2004:02:02:27 -0700] "GET /cookie_usage.php HTTP/1.0" 200 21764 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

66.249.78.79 - - [17/Oct/2004:02:03:07 -0700] "GET /product_reviews.php?products_id=499&action=notify HTTP/1.0" 302 0 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

66.249.78.79 - - [17/Oct/2004:02:03:07 -0700] "GET /cookie_usage.php HTTP/1.0" 200 21764 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

66.207.118.206 - - [17/Oct/2004:02:03:37 -0700] "GET /index.php?cPath=32_30&osCsid=472a70b19e8107dab70be0b6c3ee42e2 HTTP/1.1" 200 27133 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)"

66.207.118.206 - - [17/Oct/2004:02:04:22 -0700] "GET /product_reviews.php?products_id=443&osCsid=b8841dddbb1c72e030619851e0b84632 HTTP/1.1" 200 26970 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)"

66.249.78.54 - - [17/Oct/2004:02:04:37 -0700] "GET /index.php?cPath=1_27&page=1&sort=2a&action=buy_now&products_id=257 HTTP/1.0" 302 0 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

66.249.78.54 - - [17/Oct/2004:02:04:38 -0700] "GET /cookie_usage.php HTTP/1.0" 200 21764 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

66.249.78.9 - - [17/Oct/2004:02:04:52 -0700] "GET /index.php?cPath=1_27&page=1&sort=2a&action=buy_now&products_id=138 HTTP/1.0" 302 0 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

66.249.78.9 - - [17/Oct/2004:02:04:53 -0700] "GET /cookie_usage.php HTTP/1.0" 200 21764 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

 

You can see that googlebot does not have a session. Any idea how to prevent it from following the "action=buy_now&products_id=xyz" link?

 

Thanks, Ari