Googlebot referrer has been changed

[peterr]

Hi,

 

Just noticed this morning in this thread:

 

http://www.webmasterworld.com/forum3/24900.htm

 

Since a couple of days I was wondering why my tracking system didn't register any hits from Googlebot. Then I discovered that the referrer ID in the logs did change from "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" to "Googlebot/2.1 (+http://www.google.com/bot.html)".

 

Just to let you know since I didn't notice any tread on this ...

 

cheers Panze

 

I'm going to have to check the web server logs, and make sure that session ID's will still turn off. I'm a bit "rusty" with the osC code that checks this, and did have a small script somewhere to push the user agent/referrer details and check against the file 'spiders.txt'

 

The last thing I want for any osCommerce sites is for session ID's to turn up in Google. That said, the current entry in spiders.txt that we use is:

 

googlebot

 

and the osC code looks for any ocurrance of _that_ string in the "user agent" NOT the referrer. :D

 

It is retrieved like this in /application_top.php

 

$user_agent = strtolower(getenv('HTTP_USER_AGENT'));

 

I don't want to be an alarmist, but as they say, better safe than sorry, so I'm going to need to test what is now showing up in the logs for Google, and test it to make sure session ID's still get turned off.

 

No doubt, a 'safer' entry in spiders.txt would be just:

 

google

 

Thanks,

 

Peter

[peterr]

Hi,

 

Have found the (very rough) piece of code that I used mths back to check for 'msnbot'. You will be able to see where the user agent line/s have been commented out, so just modify that line accordingly.

 

<?php
// include the domain checking functions
 require('includes/application_top.php');
 
echo $session_started;
echo '<br>';
echo $user_agent;
echo '<br>';
echo $spider_flag;
echo '<br>';
echo $spiders[$i];
echo '<br>';
echo $SID;
echo '<br>';

// start the session
 $session_started = false;
 if (SESSION_FORCE_COOKIE_USE == 'True') {
   tep_setcookie('cookie_test', 'please_accept_for_session', time()+60*60*24*30, $cookie_path, $cookie_domain);

   if (isset($HTTP_COOKIE_VARS['cookie_test'])) {
     tep_session_start();
     $session_started = true;
   }
 } elseif (SESSION_BLOCK_SPIDERS == 'True') {
   //$user_agent = strtolower(getenv('HTTP_USER_AGENT'));
   //$user_agent = strtolower("Googlebot/2.1 (+http://www.googlebot.com/bot.html)");
   $user_agent = strtolower("msnbot/0.11 (+http://search.msn.com/msnbot.htm)");
   $spider_flag = false;

   if (tep_not_null($user_agent)) {
     $spiders = file(DIR_WS_INCLUDES . 'spiders.txt');

     for ($i=0, $n=sizeof($spiders); $i<$n; $i++) {
       if (tep_not_null($spiders[$i])) {
         if (is_integer(strpos($user_agent, trim($spiders[$i])))) {
           $spider_flag = true;
           break;
         }
       }
     }
   }

   if ($spider_flag == false) {
     tep_session_start();
     $session_started = true;
   }
 } else {
   tep_session_start();
   $session_started = true;
 }

// set SID once, even if empty
 $SID = (defined('SID') ? SID : '');

echo $session_started;
echo '<br>';
echo $user_agent;
echo '<br>';
echo $spider_flag;
echo '<br>';
echo $spiders[$i];
echo '<br>';
echo $SID;
echo '<br>';
?>

 

Peter