Spiders crawl unwanted pages, create sessions etc.

[kitchenniche]

Hi there,

 

Here's a print-screen from my who's online:

 

 

I have a few questions about it:

 

1. Yahooseeker crawls cookie_usage.php even though i have it disallowed in the robots.txt. Yahooseeker crawls the cookie_usage.php now for about 20 minutes again and again and again... any ideas?

 

2. A visitor is at: shopping_cart.php even though he/she has nothing in the cart. I know this happens if you click on shopping cart but I see this happening very often. Any ideas for this?

 

3. Yahoo still create session id's from time to time even though i have the newest spiders.txt installed... Why is that?

 

4. Some bots, especially "Yahoo" and "Fast" crawl sites like: http://www.kitchenniche.ca/product_info.ph...roducts_id=1086 even though i have the SEO-Friendly URL's installed. Any ideas for this?

 

 

Thanks so much

 

Sandra

[kitchenniche]

No ideas?

[kitchenniche]

up :-"

[Guest]

how did you get the search engine names to show up instead of guests?

 

 

i ban the search engines from unwanted pages by using robots.txt or nolink,nofollow in the meta tags

[kitchenniche]

how did you get the search engine names to show up instead of guests?

i ban the search engines from unwanted pages by using robots.txt or nolink,nofollow in the meta tags

 

Hi eww,

 

Thank you for your reply

 

To see the Search Engines I have installed this contribution: Who's Online Enhancement

 

I do have a robots.txt and it actually works for other pages but not for the cookie_usage.php .

 

Sandra

[bobg7]

I added this to the top of the .php pages to keep a yahoo bot from doing it's thing.

 

<?

$blockip = array("68.142.230.187");

$x = count($blockip);

for ($y = 0; $y < $x; $y++) {

if ($REMOTE_ADDR == $blockip[$y]) {

echo ("Due to bandwidth abuse, your IP Address has been banned from this website");

Exit;}

}

?>

 

You can add ip addresses with a comma between each one ("xxx.xxx.xxx.xxx, xxx.xxx.xxx.xxx")

And change the echo message if you want. The bot may not see it, but it's there anyways.