Strange text characters

[doon1947]

Hi,

HTTP Server: Apache/2.2.23 PHP Version: 5.2.17 (Zend: 2.2.0)

osCommerce 2.2-MS2

 

I have a site that has been working just fine for sometime. However lately some the text characters are displaying incorrectly.

 

The dash "-" character shows up as –

The apostrope "'" shows up as ’

 

See here http://www.always-with-me.com/product_info.php?cPath=21&products_id=1006

 

Can someone explain what is happening and how to fix it.

 

Al

[MrPhil]

Were any changes made? New/revised text, or change to the display encoding? If any editing, was it done in UTF-8 encoding? What you're seeing are UTF-8 apostrophes and dashes being displayed in Latin-1 (ISO-88951-1). Was your site displaying in UTF-8 before? Does anyone else have access to the site to edit files?

[doon1947]

Were any changes made? New/revised text, or change to the display encoding? If any editing, was it done in UTF-8 encoding? What you're seeing are UTF-8 apostrophes and dashes being displayed in Latin-1 (ISO-88951-1). Was your site displaying in UTF-8 before? Does anyone else have access to the site to edit files?

 

No, there were no changes made to the site. I wouldn't know how to change the coding.

[MrPhil]

2.2 MS2 is quite long of tooth, and will require many patches once your host goes to PHP 5.3 or higher.

 

The question to be answered is, "how did UTF-8 characters get into your database?" Presumably your page display has been Latin-1 (ISO-8859-1) all along (as it still is*). Do you know how to go into phpMyAdmin and look around at the database character encoding ("collation") overall, for tables, and for individual fields? If anything is now UTF-8, you should ask your host if they just upgraded/patched MySQL and might have accidentally changed any collations from latin1... to utf8... and converted the data for you. You (or someone else with admin access) didn't just cut and paste in some text that, oh, started out in Word, was cut and pasted into a UTF-8 application of some sort, and then cut and pasted into osC?

 

* Actually, my browser tells me your page is Windows-1252 (Latin-1 + Smart Quotes), even though the encoding is specified as ISO-8859-1. There's a slight possibility that your server has been broken for a long time, and was actually overriding your encoding request of Latin-1 with UTF-8 (properly displaying UTF-8 characters, such as the en-dash and single close quote/apostrophe) and was just fixed to really display Latin-1. Ask your host if nothing else pans out.

[doon1947]

Mr Phil,

 

2.2 MS2 is quite long of tooth, and will require many patches once your host goes to PHP 5.3 or higher.

 

Yes I understand 2.2 MS2 is old, however I use it on purpose because I need the contribution Option Types 2.1.3 which is not compatable with newer versions of OSC.

 

The question to be answered is, "how did UTF-8 characters get into your database?" Presumably your page display has been Latin-1 (ISO-8859-1) all along (as it still is*). Do you know how to go into phpMyAdmin and look around at the database character encoding ("collation") overall, for tables, and for individual fields? If anything is now UTF-8, you should ask your host if they just upgraded/patched MySQL and might have accidentally changed any collations from latin1... to utf8... and converted the data for you. You (or someone else with admin access) didn't just cut and paste in some text that, oh, started out in Word, was cut and pasted into a UTF-8 application of some sort, and then cut and pasted into osC?

 

Don't use PHP MyAdmin but use HeidiSQL and can look at tables/Records/Fields. Have found the following:

 

character_set_client utf8

character_set_connection utf8

character_set_database utf8

character_set_file_system binary

character_set_results utf8

character_set_server latin1

character_set_system utf8

collation_connection utf8_general_ci

collation_database utf8_general_ci

collation_server latin1_swedish_ci

Why swedish, is that an error?

 

* Actually, my browser tells me your page is Windows-1252 (Latin-1 + Smart Quotes), even though the encoding is specified as ISO-8859-1. There's a slight possibility that your server has been broken for a long time, and was actually overriding your encoding request of Latin-1 with UTF-8 (properly displaying UTF-8 characters, such as the en-dash and single close quote/apostrophe) and was just fixed to really display Latin-1. Ask your host if nothing else pans out.

 

Have asked my host, they reply:

Do you have an approximate

timeframe as to when the strange characters began appearing on the

pages? It seems unlikely that this would be related to a server

software upgrade of some kind, and have more to do with a character

encoding mismatch. Additionally, this does not seem to be occurring

uniformly in all cases. For example, there are numerous instances on

the page you've indicated where the hyphen character is displayed

correctly (eg. "234 Crystal - Mix Rainbow (3 in chevron) $2.50" under

"Embellishments"). Are you seeing these characters in the osCommerce

back-end also?

 

Thanks for your input

[MrPhil]

Why swedish, is that an error?

MySQL was originally developed in Sweden, so Latin-1 with Swedish sort collation is the default. For English language sites, you'll never see various accented characters anyway, so no blood, no foul.

 

I'll try to look at this some more tonight. I've never seen "HeidiSQL", but having "latin1" in there with all the UTF-8s worries me a bit.

 

Oh, and please use quote BBCode tags when quoting another post -- it makes it so much easier to tell what's what and who's speaking.

[MrPhil]

I need the contribution Option Types 2.1.3

Does this help? Option Types 2.3.1 http://addons.oscommerce.com/info/8193 should work with 2.3.3, or at worst, with very few changes.

 

I see that somebody or something has changed some of the text characters from UTF-8 to ASCII, but not all of them. Can I presume that you did that since you started this topic? It seems to have confused your host, who thinks that the characters are displaying correctly now. Are they aware that some have been changed?

 

I don't know what applies to what in that list from HeidiSQL, but I'm very concerned that it's not uniformly Latin-1 or UTF-8. Something has been in Latin-1 at some points, and UTF-8 at other times, to get UTF-8 text on Latin-1 pages. Did you do any editing of your pages or content cut and pasted from Word? That might account for the non-ASCII characters, but still needs an explanation for how they ended up in UTF-8, unless you're using certain browsers that convert automatically for UTF-8 pages (but have you ever displayed your pages in UTF-8?).

[doon1947]

Does this help? Option Types 2.3.1 http://addons.oscommerce.com/info/8193 should work with 2.3.3, or at worst, with very few changes.

 

I see that somebody or something has changed some of the text characters from UTF-8 to ASCII, but not all of them. Can I presume that you did that since you started this topic? It seems to have confused your host, who thinks that the characters are displaying correctly now. Are they aware that some have been changed?

 

I don't know what applies to what in that list from HeidiSQL, but I'm very concerned that it's not uniformly Latin-1 or UTF-8. Something has been in Latin-1 at some points, and UTF-8 at other times, to get UTF-8 text on Latin-1 pages. Did you do any editing of your pages or content cut and pasted from Word? That might account for the non-ASCII characters, but still needs an explanation for how they ended up in UTF-8, unless you're using certain browsers that convert automatically for UTF-8 pages (but have you ever displayed your pages in UTF-8?).

 

While in the DB I changed 2 characters back to "apostrophe" just as a test. That is the only thing I have done.

As best remember all the "apostrophe" and "dash" characters were correct when I first entered them when I built the website a year ago. Some of them have changed by themselves without any intervention on my part.

 

Viewing the DB with HeidiSQL I see where I can change the Default Collation, there are about 20 options including "latin1_swedish_ci" and a whole selection of "utf" options in various languages. There is also a checkbox whcih reads "Convert Data"

 

I'm in the dark here.

 

Al.