Easy Populate & Products Attributes

[surfalot]

I've got EP v.290b_1 installed and everything was working fine till I hit the 1000 product mark, now when I try to download I get an error message, which according to my provider is caused by it timing out. According to my provider, they have a restriction of 30 seconds script execution time on our servers, so is there anyway to work around this, or is EP only going to be good for uploading? :(

try v2.76d. I added features for downloading large databases. you can filter the download to include only certain categories or certain manufacturers. I also added a feature to download on-the-fly which may work to eliminate the timeouts even for dbs with over 8000 products. (takes over 45 minutes to download with all the attributes. Always backup before trying something new!

[surfalot]

Okay, just need to be sure of something: the above line actually says:

 

define('DIR_FS_DOCUMENT_ROOT', '/home/mythgame/public_html'); // where the pages are located on the server

 

Now should I replace that whole string, or just add the $DOCUMENT_ROOT behind the , like so:

 

define('DIR_FS_DOCUMENT_ROOT', $DOCUMENT_ROOT '/home/mythgame/public_html'); // where the pages are located on the server

 

Thanks Surf!

Cory

if the site's admin functions are working before, you don't need to change that setting. Use either this:

define('DIR_FS_DOCUMENT_ROOT', $DOCUMENT_ROOT );

-or this-

define('DIR_FS_DOCUMENT_ROOT', '/home/mythgame/public_html');

 

you need to be most concerned with the temp directory setting at the top of the ep file.

[ruudger]

Hi,

Installed V2.76D , created the temp dir mod 777, made the $DOCUMENT_ROOT change, but get only a refresh of the screen when trying to download, or one of the other export functions.

 

any help appreciated :)

 

Ruud

[MLHmptn]

How could one manipulate the easypopulate.php file to delete items via product_model instead of via products_id? I'm sure some kind of query could be inserted before or within the delete function. My vendor supplies me with a discontinued items list that supplies my product_model and sometimes its in upwards of 3000 items and manually doing it every quarter is too laborous via phpMyAdmin.

 

Here is my experimental code...it deletes the product from the products.table but doesn't delete the product from the products_description or products_to_categories table.

 

Here is my modified code:

 

//if delete
if ( $v_action == 'delete' ){
	// they want to delete this product.
	echo EASY_LABEL_DELETE_STATUS_1 . $v_products_id . EASY_LABEL_DELETE_STATUS_2 . '<br>';
	// Get the ID
	$products_query = tep_db_query("select products_id, products_model from " . TABLE_PRODUCTS);
	while ($products_array = tep_db_fetch_array($products_query)) {
		$products[$products_array['products_model']] = $products_array['products_id'];
	}
$v_products_model = $v_products_id;
$delete_id = $v_products_id;
// get category ID
	// kill in the products_to_categories
tep_db_query("delete from " . TABLE_PRODUCTS . " where products_id ='" .  $delete_id . "'");
tep_db_query("delete from " . TABLE_PRODUCTS_DESCRIPTION . " where products_id ='" .  $delete_id . "'");
tep_db_query("delete from " . TABLE_PRODUCTS_TO_CATEGORIES . " where products_id ='" .  $delete_id . "' and categories_id = '" . $v_categories_id . "' ");

 $prod_attrib_query = tep_db_query("select products_attributes_id from " . TABLE_PRODUCTS_ATTRIBUTES . " where products_id ='" .  $delete_id . "'");
while ($prod_attrib1 = tep_db_fetch_array($prod_attrib_query)) {
tep_db_query("delete from " . TABLE_PRODUCTS_ATTRIBUTES_DOWNLOAD . " where products_attributes_id  ='" .  $prod_attrib1[products_attributes_id]  . "'");
}

tep_db_query("delete from " . TABLE_PRODUCTS_ATTRIBUTES . " where products_id ='" .  $delete_id . "'");

	// Kill in the products table

	return; // we're done deleteing!

 

Here is the original code

 

//if delete
if ( $v_action == 'delete' ){
	// they want to delete this product.


	echo EASY_LABEL_DELETE_STATUS_1 . $v_products_id . EASY_LABEL_DELETE_STATUS_2 . '<br>';
	// Get the ID
	$delete_id = $v_products_id;
// get category ID	   
	// kill in the products_to_categories
tep_db_query("delete from " . TABLE_PRODUCTS . " where products_id ='" .  $delete_id . "'");
tep_db_query("delete from " . TABLE_PRODUCTS_DESCRIPTION . " where products_id ='" .  $delete_id . "'");
tep_db_query("delete from " . TABLE_PRODUCTS_TO_CATEGORIES . " where products_id ='" .  $delete_id . "' and categories_id = '" . $v_categories_id . "' ");

 $prod_attrib_query = tep_db_query("select products_attributes_id from " . TABLE_PRODUCTS_ATTRIBUTES . " where products_id ='" .  $delete_id . "'");
while ($prod_attrib1 = tep_db_fetch_array($prod_attrib_query)) {
tep_db_query("delete from " . TABLE_PRODUCTS_ATTRIBUTES_DOWNLOAD . " where products_attributes_id  ='" .  $prod_attrib1[products_attributes_id]  . "'");
}

tep_db_query("delete from " . TABLE_PRODUCTS_ATTRIBUTES . " where products_id ='" .  $delete_id . "'");

	// Kill in the products table

	return; // we're done deleteing!

 

 

Any help would be greatly appreciated!

 

Mike

[Guest]

if you want to try to use EP for deleting, you should probably do it in the steps with which you have to add them, if done so manually.

start in reverse, with the attributes, and attribute options.

then you can work back to the other tables.

[surfalot]

Hi,

Installed V2.76D , created the temp dir mod 777, made the $DOCUMENT_ROOT change, but get only a refresh of the screen when trying to download, or one of the other export functions.

 

any help appreciated :)

 

Ruud

you probably have globals turned off in your host's php. this version, and all of them as far as I know, won't work on that host. check back, I'm working on the fix.

[surfalot]

How could one manipulate the easypopulate.php file to delete items via product_model instead of via products_id? I'm sure some kind of query could be inserted before or within the delete function. My vendor supplies me with a discontinued items list that supplies my product_model and sometimes its in upwards of 3000 items and manually doing it every quarter is too laborous via phpMyAdmin.

 

Any help would be greatly appreciated!

 

Mike

why rewrite, adapt the product removal code in /admin/categories.php

search for :

case 'delete_product_confirm':

from there to the next break; statement is the product removal code. hopefully you won't have too much trouble adopting it. please post your solution, I'd like to add it to the update I'm working on. always backup and/or use a test server! ;)

[Maynard]

I mentioned a leading-quote bug in the excel-safe csv manipulation, and I wasn't going to post my fix, but then you said

why's that?

So: OK, here it is:

 

The issue is this: PHP's fgetcsv (used in excel-safe) doesn't like a csv field that begins with a double-quote.

 

For example, my product name is:

"A Song" backing track download

 

In the csv file, that becomes:

"""A Song"" backing track download"

 

and fgetcsv spits out:

A Song backing track download

 

It doesn't mind when the quotes are further into the field, it just seems to rebel against them at the beginning.

(PHP version 4.3 I think - perhaps this has been fixed in later versions?)

 

SO: we've got to do the csv parsing ourselves, without fgetcsv.

Here's my attempt, which may be buggy, but seems to work with my site:

 

around line 820, where it says:

 

		// get the entire file into an array
	$readed = file($tempdir . $localfile);
}

if ($excel_safe_output == true) {

 

replace:

  // do excel safe input
  unset($readed);					// kill array setup with above code
  $readed = array();				 // start a new one for excel_safe_output
  $fp = fopen($tempdir . ($usrfl?$usrfl_name:$localfile),'r') or die('##Can not open CSV file for reading. Script will terminate.<br />');  // open file
  while($line = fgetcsv($fp,32768,$separator))   // read new line (max 32K bytes)
  {
	unset($line[(sizeof($line)-1)]);  // remove EOREOR at the end of the array
	$readed[] = $line;				// add to array we will process later
  }
  $theheaders_array = $readed[0];	 // pull out header line
  fclose($fp);						// close file

 

with

// Maynard's CSV Fix ###################################################
// the fgetcsv doesn't like a field beginning with "", so we'll parse it ourselves
// use readed, but parse it ourselves according to $separator, and " as field encloser

// I copied this from below:
// now we string the entire thing together in case there were carriage returns in the data
  $newreaded = "";
  foreach ($readed as $read){
	$newreaded .= $read;
  }
  // now newreaded has the entire file together without the carriage returns.
  // if for some reason excel put qoutes around our EOREOR, remove them then split into rows
  $newreaded = str_replace('"EOREOR"', 'EOREOR', $newreaded);
  $readed = explode( $separator . 'EOREOR',$newreaded);
// dump the last entry - it's blank
unset($readed[sizeof($readed)-1]);	
// parse it all!
foreach ($readed as $key => $line) {
	$ppp = 0; // pointer
	$line1 = array();
	$remains = trim($line);
	while ($remains != '') {
		if (substr($remains,0,1) == '"') {
			// this is an enclosed portion
			$ppp1 = -1; // so that our first loop points to the right place in $remains
			$ppp2 = -1;
			// we've got to find the next " that isn't ""	
			while($ppp1 < strlen($remains) && $ppp1 == $ppp2 & $ppp1 !== FALSE) {
				$ppp1 = strpos($remains, '"', $ppp2+2);
				$ppp2 = strpos($remains, '""', $ppp2+2); // use ppp2 because then """ will get properly understood				
				}
			// now $ppp1 points to the closing "
			$line1[] = substr($remains,1,$ppp1-1);
			// find the next separator - for a well formed CSV file, it MUST be a separator (and not in quoted text) 
			$ppp1 = strpos($remains, $separator, $ppp1);
			$remains = ($ppp1 !== false) ? substr($remains,$ppp1+1) : ''; 			
			} else {
			// this is an un-enclosed portion - find the next separator
			$ppp1 = strpos($remains, $separator, 0);
			$line1[] = ($ppp1 !== false) ? substr($remains,0,$ppp1) : $remains;
			$remains = ($ppp1 !== false) ? substr($remains,$ppp1+1) : '';						
			}
		}
	// replace that line with our new array
	$readed[$key] = $line1;
	}
  $theheaders_array = $readed[0];	 // pull out header line	
// EOF Maynard's CSV Fix ###################################################
/* Here's the old bit commented out, in case you want it :-)
  // do excel safe input
  unset($readed);					// kill array setup with above code
  $readed = array();				 // start a new one for excel_safe_output
  $fp = fopen($tempdir . ($usrfl?$usrfl_name:$localfile),'r') or die('##Can not open CSV file for reading. Script will terminate.<br />');  // open file
  while($line = fgetcsv($fp,32768,$separator))   // read new line (max 32K bytes)
  {
	unset($line[(sizeof($line)-1)]);  // remove EOREOR at the end of the array
	$readed[] = $line;				// add to array we will process later
  }
  $theheaders_array = $readed[0];	 // pull out header line
  fclose($fp);						// close file
*/

 

Hope it helps someone.

[surfalot]

The issue is this: PHP's fgetcsv (used in excel-safe) doesn't like a csv field that begins with a double-quote.

It doesn't mind when the quotes are further into the field, it just seems to rebel against them at the beginning.

(PHP version 4.3 I think - perhaps this has been fixed in later versions?)

your logic appears solid from what I know about the Excel CSV format, thanks for that. I might include that in the next version I'm working on as an alternative to the misfiring fgetcsv function. that's too bad, that function has always been good to me in past work. It appears as if you have between PHP 4.3.4 and 4.3.9 bug report

Edited January 27, 2007 by surfalot

[Maynard]

your logic appears solid from what I know about the Excel CSV format, thanks for that. I might include that in the next version I'm working on as an alternative to the misfiring fgetcsv function. that's too bad, that function has always been good to me in past work. It appears as if you have between PHP 4.3.4 and 4.3.9 bug report

Yes - well, 4.3.10 it says. Looks like it's not fixed til 5.10.0 - and I think my online host uses a version 4.

It's a pity, fgetcsv is much neater than the code I've written!

[ruudger]

thanks Todd,

 

I've put a php.ini in the catalog directory with

register_globals = on but no difference.

 

Ruud

[Jack_mcs]

if you have v2.76d, enable the Excel Safe Output feature and the Preserve Tabs, Carriage returns and Line feeds feature. Should handle it fine. That is why I added those features.

Good job Todd. Hopefully your update will fix the mess this contribution has become. But I'm currious why you decided to go with non-standard model size? Also, are you aware of a problem with the excel friendly option? I've had several problems with various sites where the download wouldn't work. I didn't have time to trace the problem but found that by resetting the settings you have chosen to standard ones, the download works. Just wondering if you have ran across this?

 

Jack

[Guest]

Hi All,

 

Can anybody help please, I have installed easy populate and am having troubles. It all appears fine in the admin control panel , but when i try and download all it does is refresh the screen and does'nt down load any files.

 

After looking on this thread i can see somebody else having the same problem, and people have suggested waiting for a fix or new update. IS THIS TRUE, i just want to know if i am wasting my time persuing this matter and should wait for the update maybe.

 

I have carried out all the changes and still no joy.

[surfalot]

thanks Todd,

 

I've put a php.ini in the catalog directory with

register_globals = on but no difference.

 

Ruud

even if you add that, doesn't mean it's working. You can use the phpinfo() function to display the result settings for your web space.

 

So you have not added the register globals patch to your osC?

 

see post #4919 of this thread and please report the last line number that code was on when you could see the "I'm Here" displayed.

 

A couple pages later, someone did help with that. I'm betting you will have the same results as they did.

[surfalot]

Good job Todd. Hopefully your update will fix the mess this contribution has become. But I'm currious why you decided to go with non-standard model size? Also, are you aware of a problem with the excel friendly option? I've had several problems with various sites where the download wouldn't work. I didn't have time to trace the problem but found that by resetting the settings you have chosen to standard ones, the download works. Just wondering if you have ran across this?

 

Jack

the model size came from the 2.76c I started with. the update is defaulted to 12. Hallmark pointed that out a little while back. What standard settings? the version I released has the same defaults as the 2.76c I started with (I thought).

[surfalot]

Hi All,

 

Can anybody help please, I have installed easy populate and am having troubles. It all appears fine in the admin control panel , but when i try and download all it does is refresh the screen and does'nt down load any files.

 

After looking on this thread i can see somebody else having the same problem, and people have suggested waiting for a fix or new update. IS THIS TRUE, i just want to know if i am wasting my time persuing this matter and should wait for the update maybe.

 

I have carried out all the changes and still no joy.

yes it is true. I'm doing some final test and looking at a simple method to add new fields without having to mod the whole script.

[Guest]

yes it is true. I'm doing some final test and looking at a simple method to add new fields without having to mod the whole script.

 

Ok,

 

I will await your post and keep an eye out for your post

[surfalot]

Also, are you aware of a problem with the excel friendly option? I've had several problems with various sites where the download wouldn't work.

are you saying the download wouldn't work only when the excel safe output is true?

[Jack_mcs]

are you saying the download wouldn't work only when the excel safe output is true?

No, actually what I meant to say was that uploads won't work unless one of the options is reset. I don't know which since I just went in and changed them all to what I've always used, except for your new option, of course, and that I turned off. It worked after that so I didn't pay much attention to it. Then two shops after that had the same problem. Changing the settings fixes it in those too. The symptom is that when you try to upload, it appears to accept it and says it uploaded, but it isn't. None of the individual lines are printed for the update. So instead of saying something like:

model xxx ... updated

update complete

 

it justs says

update complete

 

If I have time I will take a closer look at the problem but thought you might be aware of it already.

 

Jack

[surfalot]

None of the individual lines are printed for the update. So instead of saying something like:

model xxx ... updated

update complete

 

it justs says

update complete

 

If I have time I will take a closer look at the problem but thought you might be aware of it already.

 

Jack

there has been an issue with PHP 5 and the array_walk function. but in that case it will process one and only one product.

[surfalot]

No, actually what I meant to say was that uploads won't work unless one of the options is reset. I don't know which since I just went in and changed them all to what I've always used, except for your new option, of course, and that I turned off. It worked after that so I didn't pay much attention to it. Then two shops after that had the same problem. Changing the settings fixes it in those too.

would like to know what those standard setting are for a closer look... :)

the main thing I did with the update is remove the rancid global use.

[Optimalkiller]

when i run the easypopulate.php it only seems to import the last line and ignore tge others see below for example.

below this is the output of easypopulate.php.

 

File uploaded.

Temporary filename: C:\Program Files\xampp\tmp\php66.tmp

User filename: EP2007Jan28-2311.txt

Size: 593

| None | | test | test | | | | | | | | 5 | 0 | | 28/01/2007 | 1 | | Mov | Test Test | | --none-- | Active Updated

| None | | test2 | test2 | | | | | | | | 5 | 0 | | 28/01/2007 | 1 | | Mov | Test Test | | --none-- | Active Updated

 

 

and when i select the products to see whats been imported only test2 is listed. and below is my txt file

 

v_products_model v_products_image v_products_name_1 v_products_description_1 v_products_url_1 v_products_name_2 v_products_description_2 v_products_url_2 v_products_name_3 v_products_description_3 v_products_url_3 v_products_price v_products_weight v_date_avail v_date_added v_products_quantity v_manufacturers_name v_categories_name_1 v_categories_name_2 v_categories_name_3 v_tax_class_title v_status EOREOR

None test test 5 0 28/01/2007 23:22 1 Mov Test Test --none-- Active EOREOR

None test2 test2 5 0 28/01/2007 23:22 1 Mov Test Test --none-- Active EOREOR

 

 

Can anyone yell me why the only last item is imported and the others are ignored please help!!!

[Jack_mcs]

would like to know what those standard setting are for a closer look... :)

the main thing I did with the update is remove the rancid global use.

It was just the model size and quotes that I reset, besides the four new options you added.

 

Jack

[surfalot]

when i run the easypopulate.php it only seems to import the last line and ignore tge others see below for example.

below this is the output of easypopulate.php.

 

File uploaded.

Temporary filename: C:\Program Files\xampp\tmp\php66.tmp

User filename: EP2007Jan28-2311.txt

Size: 593

| None | | test | test | | | | | | | | 5 | 0 | | 28/01/2007 | 1 | | Mov | Test Test | | --none-- | Active Updated

| None | | test2 | test2 | | | | | | | | 5 | 0 | | 28/01/2007 | 1 | | Mov | Test Test | | --none-- | Active Updated

and when i select the products to see whats been imported only test2 is listed. and below is my txt file

 

v_products_model v_products_image v_products_name_1 v_products_description_1 v_products_url_1 v_products_name_2 v_products_description_2 v_products_url_2 v_products_name_3 v_products_description_3 v_products_url_3 v_products_price v_products_weight v_date_avail v_date_added v_products_quantity v_manufacturers_name v_categories_name_1 v_categories_name_2 v_categories_name_3 v_tax_class_title v_status EOREOR

None test test 5 0 28/01/2007 23:22 1 Mov Test Test --none-- Active EOREOR

None test2 test2 5 0 28/01/2007 23:22 1 Mov Test Test --none-- Active EOREOR

Can anyone yell me why the only last item is imported and the others are ignored please help!!!

They are both imported, but the second item with model number "None" overwrites the first item with model number "None"

 

the problem is, as stated in the docs, you MUST make every model number unique. That is true for the import file and all products on your catalog.

 

Please do read through the docs, they're not the best, but most of the info you need is there.

[Optimalkiller]

great guys thank you very much all is well.

 

btw thank you for the very fast responce.