GoogleBot/others SID killer suggestion

[sfatula]

I apologize if I did not find all message related to this, I've read a few hundred. I've seen the various code changes to HTML_OUTPUT.

 

These are all fine and dandy, but, remember this gets called numerous times per page, could easily be called 50 times or more for a single web page.

 

So, I would suggest only running through the filtering logic one time per web page. This means setting a variable that says you have run the bot logic for a given run for a web page, it would be set after the first check. Next time through HTML_OUTPUT for a given web page, it would simply use the saved value to determine whether or not to append the session ID, clearly, the user agent is the same for a single page!

 

Anyway, just an idea. I've seen a few sites that timeout now and then with all the eregi loops. If you run through the logic once, an obvious performance gain.

[Ian]

Steve,

 

You are, of course right. Doing a test in html_output means you could be doing multiple tests per osc page, and adding to the page parse time.

 

I would love to say that my code passes your test as the main test is made in application_top, unfortunately I am well aware of the limitations of my code.

 

Maybe, those who wrote the other sid killers which work on user agent/ip address tests, could move these test to application top and use my $kill_sid variable.

 

This would give us a consistent api in html_output. Any one who came up with new code for testing would simply add it to application top, set the kill_sid variable and html_output would automaitically take advantage of it.

[burt]

Ian,

 

Where is your solution? Is it a contribution - I haven't seen it...

 

Cheers.

[Ian]

The code was posted as part of this thread.

 

http://www.oscommerce.com/forums/viewtopic.php...der=asc&start=0

 

As a recap. The purpose of my code was to do away with having to do multiple tests for agents/ip's for spiders.

 

It works by only enabling sid's if the customer is looged in or an item has been placed in the cart.

 

The biggest problem with the code is the buy now button, this does not rely on HTTP_POST_VARS, so a bot could follow th link, and add something to the cart and sid's would reappear.

 

The solution to this, would be to make the buy now button a form action, however this would need to be core osc code.

[burt]

The code was posted as part of this thread. <snip>

 

Cheers, I'll take a good read through that and see if I can come up with a solution...

[hobbzilla]

May I ask why the "buy now" button isn't/shouldn't be a form action and thus a core change from the GET_VARS to a POST_VARS ??

 

I think your (Ian's) solution is by far the most scaleable. It doesn't depend on ip addresses or user agent or any of the nonsense that would be impossible to keep up with. A simple check at the beginning of each page about whether or not a sid needs to be generated and used for each link (if cookies are off) is the most elegant.

 

I'm assuming that making the buy now link dependent on a form post would require a decent amount of work?

[psilun]

Why not just make the buy now buttons available only to a logged in customer.

 

I have tried the following on my test system. This is in the default.php file.

 

// effectively set PRODUCT_LIST_BUY_NOW based on customer logged in?

if (!$customer_id){

$buy_now == 0;

}else{

$buy_now == PRODUCT_LIST_BUY_NOW;

}

 

$define_list = array('PRODUCT_LIST_MODEL' => PRODUCT_LIST_MODEL,

'PRODUCT_LIST_NAME' => PRODUCT_LIST_NAME,

'PRODUCT_LIST_MANUFACTURER' => PRODUCT_LIST_MANUFACTURER,

'PRODUCT_LIST_PRICE' => PRODUCT_LIST_PRICE,

'PRODUCT_LIST_QUANTITY' => PRODUCT_LIST_QUANTITY,

'PRODUCT_LIST_WEIGHT' => PRODUCT_LIST_WEIGHT,

'PRODUCT_LIST_IMAGE' => PRODUCT_LIST_IMAGE,

'PRODUCT_LIST_BUY_NOW' => $buy_now);

 

Because the bots never log in, they will never see the link and the admin still has control of the display of the button or not.

[psilun]

Sorry. Lesson learned.

 

// set PRODUCT_LIST_BUY_NOW based on customer logged in?

if (!$customer_id){

$buy_now = 0;

}else{

$buy_now = PRODUCT_LIST_BUY_NOW;

}

 

$define_list = array('PRODUCT_LIST_MODEL' => PRODUCT_LIST_MODEL,

'PRODUCT_LIST_NAME' => PRODUCT_LIST_NAME,

'PRODUCT_LIST_MANUFACTURER' => PRODUCT_LIST_MANUFACTURER,

'PRODUCT_LIST_PRICE' => PRODUCT_LIST_PRICE,

'PRODUCT_LIST_QUANTITY' => PRODUCT_LIST_QUANTITY,

'PRODUCT_LIST_WEIGHT' => PRODUCT_LIST_WEIGHT,

'PRODUCT_LIST_IMAGE' => PRODUCT_LIST_IMAGE,

'PRODUCT_LIST_BUY_NOW' => $buy_now);

[Ian]

Ryan,

 

Actually making it a form submit is not too complicated for core oscommerce. The problem is all those wonderfull contributions which use buy now buttons.

 

What might be easier is to make the buy now function in application top redirect to a buy_now_process.php which contained the actual buy now logic. Having it on a separate page means that you then just need to add buy_now_proccess.php to your robots.txt

[Guest]

someone please come up with a solution for this...

 

for the past 4 days i have been over run with googlebots, sometimes as many as 30 instances appear in whos online, and it brings my server to a dead crawl - and they do love to drop everything into shopping carts......

 

hopefully someone can figure this out - it's nice to know that i am getting crawled and hopefully more pages are getting indexed, but it just feels like a horde of locusts has descended on the site ....

 

TIA

[dreamscape]

It works by only enabling sid's if the customer is looged in or an item has been placed in the cart.

 

AH crap it all man... I've just got done changing my feature product listings to include a "Buy Now" option and have the "Buy now" when browsing products... I was hoping to install your SID killer but see now that that won't do much good since when the googlebot comes by it will be dropping products into the shopping cart left and right...

 

what about only using SIDs if a user is logged in, or are they crucial to merging a "visitors" cart with a logged in users cart (as usually most people would only log in when going to check out)??

[jon_l]

I have the changes to handle spiders in html_output and whilst it is not efficient, it doesn't cause any noticeable delay, on my site anyway. I'll be looking to move the code in the next update to my site, but at the moment it works so I'm happy.

 

It is far more important to get this change made, anywhere, than worry about performance overheads. Especially as it can take two months or so for a spider like Google to catch up. Once it has the urls with session ids it will continue to index them for a month or two.

 

I like the look of Ian's mod, but there is the issue with buy it now.

 

Ian....Is there any plans for changes to be made to the core OSC code to resolve this issue? This mod could really do with being in the core OSC code as well, I'm sure a lot of people don't realise just how serious this issue is.

 

Jon.

[Guest]

Yes, at times I feel like trashing the whole osC out of my site :evil:

 

My position at Google from top 20 for many words has gone down like hell :(

 

I guess HPDL, Linda and Ian et all must address this burning need before any other thing is done at improvising osCommerce.

 

If there was a poll on this forum on this issue, the demand FOR making osC google friendly will stand to win 100,000 to 1.

 

Rgds

Sunny

[Waza04]

There you are:

http://www.oscommerce.com/forums/viewtopic.php?t=34729

[popstar]

While my site is not live yet, my existing page (in all of it's dull glory) typically sits on top of most search engines in my company's product class (we are listed in Yahoo, on Google the term "Sporting Goods" always places us in the top 10, "football helmets" most of the time places us #1). So my concern is that once I go live, I need to maintain the same results I have now and I am a bit afraid to change things, tho it's aparent that I have to. I've given thought to leaving all of my existing pages intact and adding java script re-directs to my soon-to-be new OSC catalog. Any thoughts on this??

 

Thanks

[popstar]

While my site is not live yet, my existing page (in all of it's dull glory) typically sits on top of most search engines in my company's product class (we are listed in Yahoo, on Google the term "Sporting Goods" always places us in the top 10, "football helmets" most of the time places us #1). So my concern is that once I go live, I need to maintain the same results I have now and I am a bit afraid to change things, tho it's aparent that I have to. I've given thought to leaving all of my existing pages intact and adding java script re-directs to my soon-to-be new OSC catalog. Any thoughts on this??

 

Thanks