Google Duplicate Content Manager version 1.0a

[♥FWR Media]

Rob,

OK my experiment seems to be working and remarkably quickly. In short, setting this contribution's mode to false on a site that is already indexed achieves the desired effect - the duplicates are removed from Google's index AND the pages left in Google's index have the title tags I want (not tags modified by the contribution).

 

Yesterday I implemented this contribution on a site which Google had indexed. Google was reporting 77 duplicate title tags for the site. I set the contribution's mode to false. This adds a "NOINDEX,FOLLOW" tag to the targetted pages.

 

Overnight Google crawled the site.

 

Today, 17 duplicated pages have now been removed from Google list of duplicate title tags - Google is now reporting only 60 duplicate title tags.

 

This is in line with what I expected based on Google's help page here : http://www.google.com/support/webmasters/b...py?answer=93710

 

The major remaing problem I have is duplications like this:

/product_info.php?cPath=6&products_id=40

‎‎/product_info.php?products_id=40‎

 

If I was to add 'cpath' to Var $getValues, would that prevent these duplicates? Can you foresee any problems with that? I think it would work and am happy to try it if someone could give me a bit of reassurance I haven't overlooked a problem with doing this ! I am a bit wary of trying this for fear of accidentally screwing up my site's ranking!

 

Kind regards

Peter

 

Thanks for feeding back Peter and glad it worked for you.

 

Regarding cPath this would not work as cPath is a valid link (category path) My suggestion for this would be ultimate seo urls as the reformatted urls do not suffer this problem.

[rochaesobrinho]

Hi Robert,

 

Thanks for the great contribution.

I have installed it and it is seems to work fine.

However, i have a question.

Today in my Google webmaster tools page appears only seven pages with duplicated meta description (my site has more than 100 products).

How should I proceed?

Should I set $IhaveDuplicateContent to TRUE and then when I perceive that the duplicated meta description no more exists on Google change it to FALSE?

[♥FWR Media]

Hi Robert,

 

Thanks for the great contribution.

I have installed it and it is seems to work fine.

However, i have a question.

Today in my Google webmaster tools page appears only seven pages with duplicated meta description (my site has more than 100 products).

How should I proceed?

Should I set $IhaveDuplicateContent to TRUE and then when I perceive that the duplicated meta description no more exists on Google change it to FALSE?

 

That is the method I initially invisaged when writing the contribution yes, so that is what I would recommend.

[tonyosc]

THANKS. it's what i need .

 

Only one problem . i have installed HeaderTags_SEO , enable robots . so I got two name="ROBOTS" , as follow :

 

====

<meta name="googlebot" content="all" />

<meta http-equiv="Content-Language" content="en-US" />

<meta name="robots" content="index, follow" />

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />

<!-- EOF: Header Tags SEO Generated Meta Tags -->

<meta name="ROBOTS" content="NOINDEX, FOLLOW" />

==================

 

 

I want to know if it's ok for robots . hope robots dont get angry .

Edited November 14, 2008 by tonyosc

[♥FWR Media]

THANKS. it's what i need .

 

Only one problem . i have installed HeaderTags_SEO , enable robots . so I got two name="ROBOTS" , as follow :

 

====

<meta name="googlebot" content="all" />

<meta http-equiv="Content-Language" content="en-US" />

<meta name="robots" content="index, follow" />

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />

<!-- EOF: Header Tags SEO Generated Meta Tags -->

<meta name="ROBOTS" content="NOINDEX, FOLLOW" />

==================

 

 

I want to know if it's ok for robots . hope robots dont get angry .

 

I would guess that it is a bad idea to have the two versions.

[DimeNote]

Hello,

 

Just to verify, when I have this installed and turned on, I should see a "<meta name="ROBOTS" content="NOINDEX, FOLLOW" />" tag on the category pages and subcategory pages as well? Example:

 

Candy (this has noindex tag)

--Chocolate (this has noindex tag)

----Snickers (this does not have noindex tag)

 

Let me know if this makes sense and if this is proper behavior for this contribution. Thanks again.

[♥FWR Media]

Hello,

 

Just to verify, when I have this installed and turned on, I should see a "<meta name="ROBOTS" content="NOINDEX, FOLLOW" />" tag on the category pages and subcategory pages as well? Example:

 

Candy (this has noindex tag)

--Chocolate (this has noindex tag)

----Snickers (this does not have noindex tag)

 

Let me know if this makes sense and if this is proper behavior for this contribution. Thanks again.

 

Read the instructions.

[DimeNote]

Read the instructions.

 

Thanks for the tip. I'm using SEO URL's so the category tree path looks different to me and I momentarily forgot that they really are just a mirror of the original www.whatever.com/index.php?cPath=x. ;)

 

Like I said, just wanted to verify. Thanks.

Edited November 21, 2008 by DimeNote

[♥clustersolutions]

Just discovered our duplicate title/tag with Google yesterday and was glad to find this contribution. We make some mods to two methods to format the output. Nothing fancy just to get the job done quickly. BTW, we use Ultimate SEO and Header Tags SEO. This is for example if you have the need to mod the title/tag output. Test, test, test, and backup, backup, backup...good luck...Tim

 

function parseMeta(){

 

$addedMeta = '';

$count = count($this->caught);

for( $i=0; $i<$count; $i++ ){

$addedMeta .= $this->caught[$i] . '_' . tep_sanitize_string(tep_output_string_protected($_GET[$this->caught[$i]])) . '-';

}

$addedMeta = rtrim($addedMeta, '-'); //*** get rid of the last '-'

$addedMeta = preg_replace('/sort_2a/', 'Sort by Name Ascending', $addedMeta); //*** customize these as you may have more sort functions than I and they are not for name, style no., and price.

$addedMeta = preg_replace('/sort_2d/', 'Sort by Name Descending', $addedMeta);

$addedMeta = preg_replace('/sort_3a/', 'Sort by Style No. Ascending', $addedMeta);

$addedMeta = preg_replace('/sort_3d/', 'Sort by Style No. Descending', $addedMeta);

$addedMeta = preg_replace('/sort_4a/', 'Sort by Price Ascending', $addedMeta);

$addedMeta = preg_replace('/sort_4d/', 'Sort by Price Descending', $addedMeta);

$addedMeta = preg_replace('/page_/', 'Page ', $addedMeta); // *** Change page_ to Page

$this->addedMeta = $addedMeta;

$this->performPCRE();

}

/**

* PCRE replace operations to inject added meta info based on matched $_GET variables

*

* @param $this->finalMeta - Sets $this->finalMeta which contains the modified meat data

*/

function performPCRE() {

 

// $pattern[] = '@<title>\s*@i'; // *** original codes

// $pattern[] = '@<meta\s*name\s*=\s*"\s*description\s*"\s*content\s*=\s*"\s*@i';

// $pattern[] = '@<meta\s*name\s*=\s*"\s*keywords\s*"\s*content\s*=\s*"\s*@i'; // *** leaving keywords alone as Google complaints only duplicate title/tag

// $replace[] = '<title>' . str_replace('-', ' | ', $this->addedMeta);

// $replace[] = '<title>' . str_replace(array('-', '_'), ' ', ucfirst($this->addedMeta));

// $replace[] = '<meta name="description" content="' . str_replace('-', ' ', $this->addedMeta);

// $replace[] = '<meta name="keywords" content="' . str_replace('-', ', ', $this->addedMeta);

 

$pattern[] = '@\s*</title>@i';

$replace[] = ', ' . str_replace(array('-'), ', ', $this->addedMeta) . '</title>'; // *** the added meta is put at the end after title.

 

$this->meta = preg_replace('@\s*"\s*>@i', ', ' . str_replace('-', ', ', $this->addedMeta) . '">', $this->meta, 1); // *** if your first meta tag is not description then this won't work for you...it puts the added meta after description. A quick fix...

 

$this->finalMeta = preg_replace($pattern, $replace, $this->meta);

}

[multiman]

Very nice job. Good contrib.

 

I've just a little problem.

 

I use STS template, SEO URL rewritting, SID Killer, Easy Meta tag.

 

so my url are like :

 

1) www.mysite.com/jeux-grande-taille-c-11.html

2) www.mysite.com/jeux-grande-taille-c-11.html?sort=3d&page=1

...

 

I've correctly install the contrib. She looks fine, I have the meta <meta name="ROBOTS" content="NOINDEX, FOLLOW" /> in the code

in the case 2) but also in the case 1)

 

After tracking, i've found that the ?sort is detected everytime (even if ?sort doesn't appear.) so I'll change your code like this

 

  var $getValues = array( 'page', 'language', 'currency');

 

instead of

 

var $getValues = array([u]'sort',[/u] 'page', 'language', 'currency');

 

It's a good way to prevent duplicate content because of the meta. But have you a idea to prevent the url in case 1) ?

 

Thank you :rolleyes:

[♥clustersolutions]

multiman, you problem is not at...

 

var $getValues = array('sort', 'page', 'language', 'currency');

 

you need to look elsewhere.

 

i also implemented a temp fix for cPath...i endup up just adding it to $getValues and use the cPath id to query the database for the category name and have the name added to $addedmeta (also written codes to exclude index.php as it is not necessary). i had more than 2000 duplicated title/description and it is down to less than half that...whew...

 

so the question now i have is, is it good to have all these pages being indexed by google, are there benefits, or is it better to turn off $IhaveDuplicateContent eventually so the number of indexed pages would decrease?

 

thank you ahead for your input...tim

[acidrush]

Hello!

 

Sorry for posting here but I do not know it where I shall turn for council in my problem!

Maybe you can help me.

 

I changed some of my product names for better results in google.

Its work's well but the the old URL works too and the google saw that's duplicate content. :huh:

 

Example:

Old URL : www.mysite.com/product-name-p-176.html

New URL : www.mysite.com/different-product-name-p-176.html

 

Different name but same product id and all URL works!

 

If I would recreate again the product with different id. I think I will loosing my postion in google results. (Now the products are in the first place in the Google results.)

 

Have you any idea please?

 

Sorry for my English!

 

Thank you

 

cheers

Edited January 17, 2009 by acidrush

[♥clustersolutions]

try this and if all else failed try a consultant...

 

fwr also has another contribution called indepent seo url validation (i supposed you also use the ultimate seo). install that and it should help with your problem as it would give google a 301 redirect of your old page to the new. your old page should eventually receive the same ranking as the old. i was in your situation before and it took about a month for the new page to receiving a ranking, and the page remained at the 1st page of google search without change. when your done with the install please be sure to validate the url response.

 

http://addons.oscommerce.com/info/5738

 

good luck...t

 

 

Hello!

 

Sorry for posting here but I do not know it where I shall turn for council in my problem!

Maybe you can help me.

 

I changed some of my product names for better results in google.

Its work's well but the the old URL works too and the google saw that's duplicate content. :huh:

 

Example:

Old URL : www.mysite.com/product-name-p-176.html

New URL : www.mysite.com/different-product-name-p-176.html

 

Different name but same product id and all URL works!

 

If I would recreate again the product with different id. I think I will loosing my postion in google results. (Now the products are in the first place in the Google results.)

 

Have you any idea please?

 

Sorry for my English!

 

Thank you

 

cheers

[multiman]

multiman, you problem is not at...

 

var $getValues = array('sort', 'page', 'language', 'currency');

 

you need to look elsewhere.

Sure, my modification is to hack my problem. If i let the 'sort' for the test, I have always the noindex meta.

For testing, I change the code :

 

  function targetsExist(){
   $caught = array();

   foreach( $this->getValues as $value ){
     if( isset($_GET[$value]) ){
     [b]var_dump ($value);
     var_dump  ($_GET[$value]);
     exit;[/b]
       $caught[] = tep_output_string_protected($value);
     }
 }
 if( !empty($caught)) {
   $this->caught = $caught;
   return true; 
 } else{
   return false;
 }
 }

 

And, even if '?sort' doesn't appear in the URL, the ?sort is detected and displayed with the dump.

I don't have any idea where the problem is, OR any idea to hack it.

 

Please feel free to help me :rolleyes:

 

i also implemented a temp fix for cPath...i endup up just adding it to $getValues and use the cPath id to query the database for the category name and have the name added to $addedmeta (also written codes to exclude index.php as it is not necessary).

 

Could you explain a little bit more your hack and could you show us your code ?

 

;)

[♥clustersolutions]

I don't even remember what I changed so I doubt I'll have the time to dig through the code to post in this forum. I was just hoping to have given some useful suggestions.

 

As for your code, change it back to the vanilla version and see if it breaks. Then do one mod at a time to see where the error lies...good luck...sorry for the late reply...T

 

 

 

Sure, my modification is to hack my problem. If i let the 'sort' for the test, I have always the noindex meta.

For testing, I change the code :

 

  function targetsExist(){
   $caught = array();

   foreach( $this->getValues as $value ){
     if( isset($_GET[$value]) ){
     [b]var_dump ($value);
     var_dump  ($_GET[$value]);
     exit;[/b]
       $caught[] = tep_output_string_protected($value);
     }
 }
 if( !empty($caught)) {
   $this->caught = $caught;
   return true; 
 } else{
   return false;
 }
 }

 

And, even if '?sort' doesn't appear in the URL, the ?sort is detected and displayed with the dump.

I don't have any idea where the problem is, OR any idea to hack it.

 

Please feel free to help me :rolleyes:

 

 

 

Could you explain a little bit more your hack and could you show us your code ?

 

;)

[kc3923]

Just discovered our duplicate title/tag with Google yesterday and was glad to find this contribution. We make some mods to two methods to format the output. Nothing fancy just to get the job done quickly. BTW, we use Ultimate SEO and Header Tags SEO. This is for example if you have the need to mod the title/tag output. Test, test, test, and backup, backup, backup...good luck...Tim

 

function parseMeta(){

 

$addedMeta = '';

$count = count($this->caught);

for( $i=0; $i<$count; $i++ ){

$addedMeta .= $this->caught[$i] . '_' . tep_sanitize_string(tep_output_string_protected($_GET[$this->caught[$i]])) . '-';

}

$addedMeta = rtrim($addedMeta, '-'); //*** get rid of the last '-'

$addedMeta = preg_replace('/sort_2a/', 'Sort by Name Ascending', $addedMeta); //*** customize these as you may have more sort functions than I and they are not for name, style no., and price.

$addedMeta = preg_replace('/sort_2d/', 'Sort by Name Descending', $addedMeta);

$addedMeta = preg_replace('/sort_3a/', 'Sort by Style No. Ascending', $addedMeta);

$addedMeta = preg_replace('/sort_3d/', 'Sort by Style No. Descending', $addedMeta);

$addedMeta = preg_replace('/sort_4a/', 'Sort by Price Ascending', $addedMeta);

$addedMeta = preg_replace('/sort_4d/', 'Sort by Price Descending', $addedMeta);

$addedMeta = preg_replace('/page_/', 'Page ', $addedMeta); // *** Change page_ to Page

$this->addedMeta = $addedMeta;

$this->performPCRE();

}

/**

* PCRE replace operations to inject added meta info based on matched $_GET variables

*

* @param $this->finalMeta - Sets $this->finalMeta which contains the modified meat data

*/

function performPCRE() {

 

// $pattern[] = '@<title>\s*@i'; // *** original codes

// $pattern[] = '@<meta\s*name\s*=\s*"\s*description\s*"\s*content\s*=\s*"\s*@i';

// $pattern[] = '@<meta\s*name\s*=\s*"\s*keywords\s*"\s*content\s*=\s*"\s*@i'; // *** leaving keywords alone as Google complaints only duplicate title/tag

// $replace[] = '<title>' . str_replace('-', ' | ', $this->addedMeta);

// $replace[] = '<title>' . str_replace(array('-', '_'), ' ', ucfirst($this->addedMeta));

// $replace[] = '<meta name="description" content="' . str_replace('-', ' ', $this->addedMeta);

// $replace[] = '<meta name="keywords" content="' . str_replace('-', ', ', $this->addedMeta);

 

$pattern[] = '@\s*</title>@i';

$replace[] = ', ' . str_replace(array('-'), ', ', $this->addedMeta) . '</title>'; // *** the added meta is put at the end after title.

 

$this->meta = preg_replace('@\s*"\s*>@i', ', ' . str_replace('-', ', ', $this->addedMeta) . '">', $this->meta, 1); // *** if your first meta tag is not description then this won't work for you...it puts the added meta after description. A quick fix...

 

$this->finalMeta = preg_replace($pattern, $replace, $this->meta);

}

 

This worked great for my title tags but when I uncomment the lines

 

$pattern[] = '@<meta\s*name\s*=\s*"\s*description\s*"\s*content\s*=\s*"\s*@i';

$replace[] = '<meta name="description" content="' . str_replace('-', ' ', $this->addedMeta);

 

to include the extra info in the descripton tag I get

 

<meta name="description" content="Sort by Price Ascending Page 1Toys & Games -" />

 

Any ideas on how to get it to read

 

<meta name="description" content="Sort by Price Ascending, Page 1, Toys & Games" />

 

Any suggestiions would be appreciated. Thanks for this great contribution and the modification code.

 

Chipmunk.

[♥clustersolutions]

try leaving that commented out...there was a reason for that and i can't remember...what's the output if the two lines were commented out? good luck...tim

 

This worked great for my title tags but when I uncomment the lines

 

$pattern[] = '@<meta\s*name\s*=\s*"\s*description\s*"\s*content\s*=\s*"\s*@i';

$replace[] = '<meta name="description" content="' . str_replace('-', ' ', $this->addedMeta);

 

to include the extra info in the descripton tag I get

 

<meta name="description" content="Sort by Price Ascending Page 1Toys & Games -" />

 

Any ideas on how to get it to read

 

<meta name="description" content="Sort by Price Ascending, Page 1, Toys & Games" />

 

Any suggestiions would be appreciated. Thanks for this great contribution and the modification code.

 

Chipmunk.

[tfbpa]

I am using Ultimate SEO Urls 2.6 together with the fix from portafixe, since I have a website which targets 7 languages and all languages should be indexed.

 

The problem with that fix is that there now is duplicate content with my default urls (english) and ..../?language=en

 

I found this addon, google duplicate content manager, which is very helpful for not indexing unwanted pages and was wondering if it is possible to specify that ONLY pages with ....?language=en have a noindex meta tag, while all other languages will still be indexed.

 

Any ideas?

 

Thanks.

[DimeNote]

I'm using Master Products contribution and I just noticed that on any master product listing that has slaves the <meta name="ROBOTS" content="NOINDEX, FOLLOW" /> tag is being added.

 

Is this what is supposed to happen? Has anyone had this same experience?

[morehawes]

Just wanted to say a BIG thanks for this! Very nice :D

Edited March 16, 2009 by morehawes

[misillsam]

Very very interesting question, i seek the solution me to.

Somebody can help to resolve this issue ?

thank you

 

 

This worked great for my title tags but when I uncomment the lines

 

$pattern[] = '@<meta\s*name\s*=\s*"\s*description\s*"\s*content\s*=\s*"\s*@i';

$replace[] = '<meta name="description" content="' . str_replace('-', ' ', $this->addedMeta);

 

to include the extra info in the descripton tag I get

 

<meta name="description" content="Sort by Price Ascending Page 1Toys & Games -" />

 

Any ideas on how to get it to read

 

<meta name="description" content="Sort by Price Ascending, Page 1, Toys & Games" />

 

Any suggestiions would be appreciated. Thanks for this great contribution and the modification code.

 

Chipmunk.

Edited April 4, 2009 by misillsam

[jester420247sl]

I am trying to install this but i am having a little difficulty. I am using the "header tags for novices" contribution along with "ultimate seo urls". My meta info is as follows:

 

<title><?php echo ((strlen($categories['categories_name']) > 1) || (strlen($manufacturers['manufacturers_name']) > 1)) ? $header_tags['products_name'] . ' - ' . $breadcrumb_tags->trail_tags(' ') . ' - ' . TITLE : TITLE; ?>
</title>
<meta name="description" content="<?php echo $header_tags['products_name']; ?>">
<meta name="keywords" content="<?php echo $header_tags['products_name']; ?>">

 

I just don't know where to add the:

ob_start();

 

and

 

$preventDuplicates->checkTarget(ob_get_clean());
echo $preventDuplicates->finalMeta . "\n";

 

any help will be greatly appreciated

[qwertyjjj]

I'm not sure I really understand how that add on will create less indexed pages.

It seems like it just rewrites the meta tags, eg

<title>currency_USD | xxxxx - xx based proxy server</title>

<meta name="description" content="currency_USD xx IP address xx server." />

 

Won't that just let google create a new title page called currency_USD in addition to all the current pages it creates?

 

At present I have over 300 listings in google with currency, page, sort, etc.

Edited September 26, 2009 by qwertyjjj

[katmars]

Thanks for your post that's great contribution,

 

 

$IhaveDuplicateContent = false;

 

Meta become : <meta name="robots" content="noindex, follow">

 

 

Does anyone of you has an idea on what need to be modified or added to this contribution in order to get 404 not found pages instead of meta modified pages as stated here above for duplicated pages ?

 

Many thanks in advance.

 

Sylvain

[marcot]

Just discovered our duplicate title/tag with Google yesterday and was glad to find this contribution. We make some mods to two methods to format the output. Nothing fancy just to get the job done quickly. BTW, we use Ultimate SEO and Header Tags SEO. This is for example if you have the need to mod the title/tag output. Test, test, test, and backup, backup, backup...good luck...Tim

 

function parseMeta(){

 

$addedMeta = '';

$count = count($this->caught);

for( $i=0; $i<$count; $i++ ){

$addedMeta .= $this->caught[$i] . '_' . tep_sanitize_string(tep_output_string_protected($_GET[$this->caught[$i]])) . '-';

}

$addedMeta = rtrim($addedMeta, '-'); //*** get rid of the last '-'

$addedMeta = preg_replace('/sort_2a/', 'Sort by Name Ascending', $addedMeta); //*** customize these as you may have more sort functions than I and they are not for name, style no., and price.

$addedMeta = preg_replace('/sort_2d/', 'Sort by Name Descending', $addedMeta);

$addedMeta = preg_replace('/sort_3a/', 'Sort by Style No. Ascending', $addedMeta);

$addedMeta = preg_replace('/sort_3d/', 'Sort by Style No. Descending', $addedMeta);

$addedMeta = preg_replace('/sort_4a/', 'Sort by Price Ascending', $addedMeta);

$addedMeta = preg_replace('/sort_4d/', 'Sort by Price Descending', $addedMeta);

$addedMeta = preg_replace('/page_/', 'Page ', $addedMeta); // *** Change page_ to Page

$this->addedMeta = $addedMeta;

$this->performPCRE();

}

/**

* PCRE replace operations to inject added meta info based on matched $_GET variables

*

* @param $this->finalMeta - Sets $this->finalMeta which contains the modified meat data

*/

function performPCRE() {

 

// $pattern[] = '@<title>\s*@i'; // *** original codes

// $pattern[] = '@<meta\s*name\s*=\s*"\s*description\s*"\s*content\s*=\s*"\s*@i';

// $pattern[] = '@<meta\s*name\s*=\s*"\s*keywords\s*"\s*content\s*=\s*"\s*@i'; // *** leaving keywords alone as Google complaints only duplicate title/tag

// $replace[] = '<title>' . str_replace('-', ' | ', $this->addedMeta);

// $replace[] = '<title>' . str_replace(array('-', '_'), ' ', ucfirst($this->addedMeta));

// $replace[] = '<meta name="description" content="' . str_replace('-', ' ', $this->addedMeta);

// $replace[] = '<meta name="keywords" content="' . str_replace('-', ', ', $this->addedMeta);

 

$pattern[] = '@\s*</title>@i';

$replace[] = ', ' . str_replace(array('-'), ', ', $this->addedMeta) . '</title>'; // *** the added meta is put at the end after title.

 

$this->meta = preg_replace('@\s*"\s*>@i', ', ' . str_replace('-', ', ', $this->addedMeta) . '">', $this->meta, 1); // *** if your first meta tag is not description then this won't work for you...it puts the added meta after description. A quick fix...

 

$this->finalMeta = preg_replace($pattern, $replace, $this->meta);

}

 

what if the url are written like this: http://www.mydomain.com/index.php/cPath/21_71/page/1/sort/2a

I dont see sort and or page in title.

 

but if i manualy change url by: http://www.mydomain.com/index.php/cPath/21_71/?page=2&sort=2a

I see the sort en page in title.

 

Any options that it works in the first url write?