sloppy words cleaner header tag versus hook

[♥raiwa]

Hello folks,

I'm thinking about an update for the Sloppy Words Cleaner

Just for those who do not know it:

What does this contrib do?
Like the title says... cleans up the sloppy inputted words by customers on their registration, edit account etc.
I also added a all lowercase for the email strings. explain sloppy input?
OK, new customer creates a new account as:
JOHN DoEskY
123 noWHERe ave.
canton, NY 12345
email: [email protected]
this will be cleaned (and stored in your db) and displayed on all invoices/screen info-text as:
John Doesky
123 Nowhere Ave.
Canton, NY 12345
email: [email protected]
I also added exception words (you can add or remove these exception words to your liking!)
i.e. "PO" for PO Box 1234 (allowing a capital "O" after the capital "P")
"IV" for John Doe IV (instead of John Doe Iv)
"RR 2" for Rural Route 2 (instead of Rr 2)
Also includes exception arrays for Names like: "Mc" and "Mac", "De la Rosa" etc.: JOHN MACDONALD  will get John MacDonald

 

I plan to modularize it for 2.3.4 BS community version and avoid core file changes as much as possible.

Now my doubt is to realize this with a header tag ot hook:

Hook:

+ fits in just in the right place on each page to filter the data before saved in the database
+ no extra database queries
- core file changes (hook call) on each page where it should be applied (create_account.php, account_edit.php, checkout_shipping_address.php, checkout_payment_address.php)
- additional configuration entries, install function in hook, but uninstall not so smooth (remove hook then set uninstall to "true" etc.)

Header Tag:

+ no core file changes, upload header tag, install, serves for all pages via selectable page list
+ all configuration entries, custom exception lists etc integrated in header tag module
- header tag is called in template_top when data is already written to database -> needs to be used in the next page (create_account_success.php for create_account.php) and read all data again out of database -> apply filter function -> write back into database. Ergo additional queries-> slows down the process.

Header tag with javascript filter on the fly? not so happy with it

Any opinion or coment very welcome

 

rgds

Rainer

[burt]

Difficult one.

My feeling is that whatever the customer inputs is correct and should be respected.

 

[♥raiwa]

Many store owners use it already who do not like to print these kind of address labels:

JOHN DoEskY
123 noWHERe ave.
canton, NY 12345

I agree for company names, there is all allowed, but for name, address, e-mail there are common rules for upper/lower case :-)

[burt]

Remember, Header Tags [unfortunate name] can be loaded into footer_scripts

[♥raiwa]

OK, but it's the same problem, the data is already processed and written into database.

create_account.php:

//filter should be applied here to all customer variables

      tep_db_perform(TABLE_CUSTOMERS, $sql_data_array);

      $customer_id = tep_db_insert_id();

...

      tep_db_perform(TABLE_ADDRESS_BOOK, $sql_data_array);


...
// header tags are loaded here
  require('includes/template_top.php');

Or do you refer to manipulate the input field contents before form submit, how? javascript?

Edited June 22, 2017 by raiwa

[burt]

Hook it is then.  It's a good use of Hooking.

Bear in mind that in the English language, getting it 100% correct is difficult;

McDonald
MacDonald
Macdonald
Mcdonald 

All are valid surnames.  In addition "Macdonald" can also be a given (first) name!

Edited June 22, 2017 by burt

[♥raiwa]

Will see if these exceptions can be all built in, thanks.

There are already separate exception arrays included for first name, second name and other data.

They are just hardcoded in the function file until now and I'll move them to user editable configuration settings.

Edited June 22, 2017 by raiwa

[frankl]

Don't forget

 

d'Argent

de Beauvoir

du Plessix

van Beethoven

von Strohm

 

then you have

De Quincey

Von Braun

La Rochefoucauld

 

and those are just surnames! You also have street names, names of cities, towns, and villages, even companies.

 

What if they work at the company called nVidia?

 

OK, I'm kind of teasing, but it's a bit of a minefield.

Perhaps use javascript to capitalize words as customers are entering them in the input fields, and if it's not right they can change it?

 

 

[♥raiwa]

@frankl,

 

Thanks for your suggestions.

This add-on exists since years and has already included extended exception lists.

  // exceptions to standard case conversion
   if ($is_name) {
       $all_uppercase = '';
       $all_lowercase = 'Or|And';
   } else {
   	 // address abreviations and anything else
       $all_uppercase = 'Aly|Anx|Apt|Ave|Bch|Blvd|Bldg|Bp|bp|Bsmt|Byu|Ch|Cors|Cswy|Cr|Crk|Crt|Cts|Cv|Cvs|Est|Ests|Expy|Frnt|Fl|Frks|Fwy|Gdn|Gtwy|Hbr|Hbrs|Hts|Hwy|Ii|Iii|Iv|Jct|Jcts|Lk|Lks|Ln|Ldg|Mnt|Mnr|Mnrs|Msn|Mtwy|Mtn|Mtns|Ne|Nw|Pkwy|Pl|Pln|Plns|Ph|Po|Pob|P.o.b.|P.O.b.|p.O.b.|p.o.B.|p.O.B.|Rm|Rr|Se|Skwy|Smt|Sw|Sta|Ste|Sq|Ter|Tpke|Trpk|Trlr|Trl|Trwy|Vl|Vlg|Vlgs|Vly|Vlys|Vi|Vii|Viii|Xi|Xing|Xrd';
       $all_lowercase = 'À|A|And|As|Am|An|Au|Aux|By|D|Da|De|Des|Del|Du|Der|Die|Das|En|Et|In|L|Le|La|Les|Of|Or|Ou|Sous|Sur|To|Von|Y';
   }



// exceptions for names and surnames
	if ($is_name) {
       $all_uppercase = '';
       $all_lowercase = 'D|De La|Da|De Las|Del|De Los|Der|Van De|Van Der|Vit De|Von|Or|And|Y|En|De|La|Del|Do|Du|Am|An|Der|Die|Das';
   } else {
       $all_uppercase = '';
       $all_lowercase = 'A|And|As|By|In|Of|Or|To';
   }
   
   $prefixes = "'|Mc|Mac|l'|L'";
   $suffixes = "'S";

	// replace target words
    $word = array ("Saint" => "St","Sainte" => "Ste","Saints" => "Sts", "Saintes" => "Stes");
	return strtr($str,$word);

 

To cover the allowed variation examples given by burt and you, I'll add now an array with terms which will not be altered by the filter function:

$exception_terms = 'McDonald|MacDonald|Macdonald|Mcdonald|De Quincey|La Rochefoucauld';

I can't confirm the third case "Von Braun". The most famous reresentative is spelled "Wernher von Braun", there may be other examples, but as always, 100% is impossible, but most store owners will prefer to get 99.9% clean data and 0.1 % or less with these kind of errors.

As already stated, all of these lists are in the old version hardcoded in the function file, but will now be editable in admin as configuration settings. So each store owner can adapt them to his local needs and language.

I can say that I'm using this since many years on my wife's store, and after adding some spanish and german exceptions which were not included in the original add-on, we didn't have to correct any data since years. Other users reported the same.

For on the fly javascript filtering:

- it would be quite a long script and slow down page load
- if a customer stops typing before he finished, and clicks out of the field, filter function would be triggered and he may get unexpected results and may get confused
- if the correction is done on the fly, even they finished typing, they also may get confused and not understand what happened
- create account is a sensitive process mostly done for checkout. I won't like to confuse any customer during this process
- once they are registered and see their corrected data it's less problematic and they will understand it easier

tks a lot for your time and suggestions

rgds

Rainer

[♥raiwa]

@frankl,

 

Forgot to mention:

For Company names any spelling is allowed, so no filtering applies to the company field.

Each entrance can be treated by different filter funtions.

[♥raiwa]

Just to remind: this topic is header tag versus hook for the new modularized version

If you wish to see what the existing add-on already does, you are invited to download it and try:

Sloppy Words Cleaner

[♥14steve14]

@raiwa

 

I already use this addon, and it has saved countless hours in getting customer details and addresses almost 100% correctly formatted.

Whether it would be better as a hook or header tag I have no real preference, but a header tag may be easier with all the core pages that the hook code would need adding to. Also having less core code changes may make upgrading the store easier in the future, unless it is possible to alter the core files required for a change automatically, which i very much doubt.

[♥raiwa]

@14steve14

Thanks Steve, the big minus for header tag is that it loads after the data is already processed and entered in the database:

https://www.oscommerce.com/forums/topic/411212-sloppy-words-cleaner-header-tag-versus-hook/?do=findComment&comment=1757857

22 hours ago, raiwa said:

Header Tag:

- header tag is called in template_top when data is already written to database -> needs to be used in the next page (create_account_success.php for create_account.php) and read all data again out of database -> apply filter function -> write back into database. Ergo additional queries-> slows down the process.

So my idea was now to use hook together with a custom group module "store" which I already use in several other add ons.

It would allow to integrate an installation process which can add the few lines of additional code, (hook register and calls) in the related core files.

On uninstall theses mods can be also auto removed. This is actually what is planned for the future 2.4 Apps to do.

This "store" module can also hold all the configuration entries, exception arrays etc.

 

rgds

Rainer