password not working after moving from latin1 to unicode8 for French Characters

[nasirmunir]

I am in the process of converting character set from iso-8859-1 to utf-8.

 

All is cool other than the password field. French characters for example are displayed properly on all pages.

 

For password field anything in the ASCII range (0 to 127) is good, and I can login. The login however, does not work with the password (that has French characters). Just to check, if I convert the char set back to ISO from UTF, I can login.

 

I have noticed that the password hash function is creating different value for both character set, and when it tries to match the password from the database, it fails. The first 12 characters are the same, and the appended characters are being created differently for ISO and UTF-8. This behaviour is only for ASCII code great than 127, in my case French Characters for example. It works fine with English characters, numbers, dollar sign, ampersand, quotes, etc. and all that falls within 127.

 

Database, and code both are changed to utf-8.

 

Is there any fix for that ?

[MrPhil]

Every password with non-ASCII characters (such as accented "French" characters) will need to be changed by the customer, using the "lost password" function. I can't think of any way to recover the old passwords and convert them (and there should never be a way to do so). I would put a notice on your front page explaining what's going on and apologizing for the inconvenience.

 

In UTF-8, accented (non-ASCII) characters are multibyte, whereas in Latin-1 they are single bytes. The password hash function would see them as different passwords typed in (and the UTF-8 is longer).

 

Changing the existing character encoding on any site and any application is always a pain, for this very reason (if none else). A good argument for always starting out with UTF-8 in the first place!