Easy Populate & Products Attributes

[surfalot]

What's the most number of lines anyone's been able to use Easy Populate to import at once?

 

I think my record is about 18,000... which, given that I have a 50,000 line .csv to import, doesn't help.

I've been trying to get some community support on that for months to no avail. But, anyway.

 

I'm curious to just -know- if anyone's been able to clear that 50,000 plateau, or something significantly larger?

that depends on your host and what you can get away with. So you can't split that into 3 and process 3 batches?

if you host will allow you to override PHP script run limit with the set_time_limit setting, the sky is the limit.

[winterion]

that depends on your host and what you can get away with. So you can't split that into 3 and process 3 batches?

if you host will allow you to override PHP script run limit with the set_time_limit setting, the sky is the limit.

 

My host (Bluehost) permits you to put in your own php.ini file, so I've done so.

(I wasn't sure where to put it, so I put a copy in public_html, a copy in my osCommerce root dir, and a copy in the admin directory. They are all identical files.)

 

I've never heard of the set_time_limit variable.

I've adjusted the following in php.ini:

 

max_execution_time = 7200 ; Maximum execution time of each script, in seconds (30)

max_input_time = 7200 ; Maximum amount of time each script may spend parsing request data (60)

memory_limit = 256M ; Maximum amount of memory a script may consume (32MB)

 

And, I've adjusted the following in easypopulate.php:

(the server does not run in safe mode)

 

set_time_limit(0);

define ('EP_SPLIT_MAX_RECORDS', 10000);

 

I've even edited the function/sessions.php file:

$SESS_LIFE = 28800;

 

So, honest to goodness, I'm stumped.

 

Edit: Forgot to add this:

EP vers: 2.76f-MS2 r1

Temp Dir:

/home/gameparl/public_html/catalog/temp/

Temp Dir is Writable

Magic Quotes is: off

register_globals is: off

Split files on: 10000 records

Model Num Size: 24

Price with tax: false

Calc Precision: 2

Replace quotes: false

Field seperator: comma

Excel safe output: true

Preserve tab/cr/lf: false

Category depth: 24

Enable attributes: false

SEF Froogle URLS: false

More Pics: false

Unknown Pics: false

HTC: false

SPPC: false

Extra Fields: false

Edited March 20, 2008 by winterion

[surfalot]

My host (Bluehost) permits you to put in your own php.ini file, so I've done so.

(I wasn't sure where to put it, so I put a copy in public_html, a copy in my osCommerce root dir, and a copy in the admin directory. They are all identical files.)

 

I've never heard of the set_time_limit variable.

I've adjusted the following in php.ini:

 

max_execution_time = 7200 ; Maximum execution time of each script, in seconds (30)

max_input_time = 7200 ; Maximum amount of time each script may spend parsing request data (60)

memory_limit = 256M ; Maximum amount of memory a script may consume (32MB)

 

And, I've adjusted the following in easypopulate.php:

(the server does not run in safe mode)

 

set_time_limit(0);

define ('EP_SPLIT_MAX_RECORDS', 10000);

 

I've even edited the function/sessions.php file:

$SESS_LIFE = 28800;

 

So, honest to goodness, I'm stumped.

 

Edit: Forgot to add this:

EP vers: 2.76f-MS2 r1

Temp Dir:

/home/gameparl/public_html/catalog/temp/

Temp Dir is Writable

Magic Quotes is: off

register_globals is: off

Split files on: 10000 records

Model Num Size: 24

Price with tax: false

Calc Precision: 2

Replace quotes: false

Field seperator: comma

Excel safe output: true

Preserve tab/cr/lf: false

Category depth: 24

Enable attributes: false

SEF Froogle URLS: false

More Pics: false

Unknown Pics: false

HTC: false

SPPC: false

Extra Fields: false

your best bet is to split the import file if you can't force the server to process the whole file. how long does it take to process 18,000 records? does it stop on the same one each time?

[winterion]

your best bet is to split the import file if you can't force the server to process the whole file. how long does it take to process 18,000 records? does it stop on the same one each time?

 

The problem we have with splitting the import file is logistical.. I have found by splitting it into 10 files, 5000 or so each, we can reliably upload the whole thing.

 

But, that means, every half-hour I need whoever is working the store to perform the next step.

 

The hope was that whoever is working, once a week, can simply start this process up, and that it would go, start to finish, on its own. :blush:

 

Or, is there an automated method in which EP can perform one .csv file in a series of uploads? (would you need Row 1 repeated, say, every 5000 lines? Could do that with an Excel macro, for example.)

[surfalot]

The problem we have with splitting the import file is logistical.. I have found by splitting it into 10 files, 5000 or so each, we can reliably upload the whole thing.

 

But, that means, every half-hour I need whoever is working the store to perform the next step.

 

The hope was that whoever is working, once a week, can simply start this process up, and that it would go, start to finish, on its own. :blush:

 

Or, is there an automated method in which EP can perform one .csv file in a series of uploads? (would you need Row 1 repeated, say, every 5000 lines? Could do that with an Excel macro, for example.)

so why not use the split function of EP? and why not answer the couple questions in the last post?

[winterion]

so why not use the split function of EP? and why not answer the couple questions in the last post?

 

Oh, I'm really sorry, big gaffe on my part. :-"

 

Umm, the 18,000 or so takes somewhere around 15 minutes? Maybe less.. I could time it exactly.

 

I'm totally new to EP, so I'm not exactly familiar how the 'split' function works. I can try and look it up, try to learn, won't beg you guys to hold my hand through that. Should it be expected that a 50,000 line .csv file would have to use the split function?

 

As for the same one each time? No, it doesn't.. different line each time. So I don't think the stop is related to the .csv file itself, like, a bad line in it.

 

Edit: Okay, my idiocy knows no bounds. The split thing was really easy. But, those five files it created (since my split was at 10k items).. any way to command EP to import those one at a time, all five, or do I have to keep coming back to the PC to do them one at a time? (Sort of a queue thing..)

Edited March 21, 2008 by winterion

[surfalot]

man, this board software is going to the dogs!! I have to clear out the emot icons before posting a reply now or everything else is cut off! :angry:

Oh, I'm really sorry, big gaffe on my part.

 

Umm, the 18,000 or so takes somewhere around 15 minutes? Maybe less.. I could time it exactly.

 

I'm totally new to EP, so I'm not exactly familiar how the 'split' function works. I can try and look it up, try to learn, won't beg you guys to hold my hand through that. Should it be expected that a 50,000 line .csv file would have to use the split function?

 

As for the same one each time? No, it doesn't.. different line each time. So I don't think the stop is related to the .csv file itself, like, a bad line in it.

 

Edit: Okay, my idiocy knows no bounds. The split thing was really easy. But, those five files it created (since my split was at 10k items).. any way to command EP to import those one at a time, all five, or do I have to keep coming back to the PC to do them one at a time? (Sort of a queue thing..)

OK once more...

so it would have been a good idea to have read through the manual like the install instructions begged you to, ey? >_< There is a small section about large files.

 

Its not expected to have to split a large file, it depends on what your host will let you get away with. I'd say it is at your host.

 

yes you have to run each one manually. If anyone has a suggestion for that, I'm all ears. I haven't put much thought into it.

Edited March 21, 2008 by surfalot

[veni]

Hello guys,

 

I have a site that is very simple example, no prices are listed. I wish to remove some of the column headers in Easy Populate, these items are :

 

v_products_name_2

v_products_url_2

v_products_description_2

v_products_name_3

v_products_url_3

v_products_description_3

v_products_price

v_products_quantity

v_products_weight

v_date_avail

v_date_added

v_categories_image_1

v_categories_name_2

v_categories_image_2

v_categories_name_3

v_categories_image_3

v_categories_name_4

v_categories_image_4

v_categories_name_5

v_categories_image_5

v_categories_name_6

v_categories_image_6

v_categories_name_7

v_categories_image_7

v_tax_class_title

 

Is is possible to remove these Items as I don't use them in my site at all?

[surfalot]

Hello guys,

 

I have a site that is very simple example, no prices are listed. I wish to remove some of the column headers in Easy Populate, these items are :

 

Is is possible to remove these Items as I don't use them in my site at all?

 

yes, if you don't need to change a bit of data, you don't need to include the column. exception would be if you are using some earlier versions that if you include any of the product name, description or URL, you must include all three or you could loose data. always backup, give it a try.

[roberovsky]

has anyone make easy populate make work for the downloading files

[telefonica600]

Ok, so I have been using easypopulate to upload the 2500 items in my store.

The contrib is GREAT!

 

I want to do a mass price change.

I can edit my .CSV file to include the price change, but I can not get it to upload.

 

Will easy populate do an update of existing items? The only way that I have been able to change it is to delete the category and reload it.

The problem there is that I need to reload all of the pictures after I delete the category. It turns into a LOT of FTP time.

 

Any suggestions?

 

Steve

 

www.reddogenterprises.net

Edited March 23, 2008 by telefonica600

[surfalot]

has anyone make easy populate make work for the downloading files

someone uploaded a version for a specific download contribution, not the standard download product built in. Other then that, not yet.

[surfalot]

Ok, so I have been using easypopulate to upload the 2500 items in my store.

The contrib is GREAT!

 

I want to do a mass price change.

I can edit my .CSV file to include the price change, but I can not get it to upload.

 

Will easy populate do an update of existing items? The only way that I have been able to change it is to delete the category and reload it.

The problem there is that I need to reload all of the pictures after I delete the category. It turns into a LOT of FTP time.

 

Any suggestions?

 

Steve

 

www.reddogenterprises.net

EP vers?

[telefonica600]

EP vers?

 

 

easypopulate.php,v 3.00 (Based on 2.78, 2.80 and 2.81b)

 

This is the only one that I have successfully been able to get to work. I am not sure why, but I can only import from the temp directory.

I can not use this to export or write to the temp directory.

I have gotten farther with this version than any other version. I have successfully loaded almost 3000 products at once.

 

Thanks for the help! I appreciate the time.

[surfalot]

easypopulate.php,v 3.00 (Based on 2.78, 2.80 and 2.81b)

 

This is the only one that I have successfully been able to get to work. I am not sure why, but I can only import from the temp directory.

I can not use this to export or write to the temp directory.

I have gotten farther with this version than any other version. I have successfully loaded almost 3000 products at once.

Thanks for the help! I appreciate the time.

what problem did you have with v2.76f or v2.76g ?

[mark27uk3]

Hi Guys,

 

I hope someone can point me in the right direction here as I have read and read and read and googled and googled (you get the picture)

 

Either I am being extremely dense or just plain blind.

 

I am pretty sure that my problem is a php5 and can find others with the same problem but no resolution. I am using EP 2.76f

 

Warning: array_merge() [function.array-merge]: Argument #1 is not an array in /home/mydomain/public_html/admin/easypopulate.php on line 577

 

Warning: array_merge() [function.array-merge]: Argument #2 is not an array in /home/mydomain/public_html/admin/easypopulate.php on line 577

 

This is line 577

 

$row2 =  array_merge($row2, tep_db_fetch_array($result2));

 

Hoping that someone else has had this issue and resolved it.

 

Thanks

 

Mark

[surfalot]

Hi Guys,

 

I hope someone can point me in the right direction here as I have read and read and read and googled and googled (you get the picture)

 

Either I am being extremely dense or just plain blind.

 

I am pretty sure that my problem is a php5 and can find others with the same problem but no resolution. I am using EP 2.76f

 

 

 

This is line 577

 

$row2 =  array_merge($row2, tep_db_fetch_array($result2));

 

Hoping that someone else has had this issue and resolved it.

 

Thanks

 

Mark

you have certainly done the prereq searching. how about offering up the info asked for in the "how to ask for help" section of the docs.

[mark27uk3]

you have certainly done the prereq searching. how about offering up the info asked for in the "how to ask for help" section of the docs.

 

ok now I am being blind which part of the docs would that be (now I feel :blush: )

 

Mark

[surfalot]

ok now I am being blind which part of the docs would that be (now I feel :blush: )

 

Mark

 

the section "XII. SUPPORT" of the manual. The "How to ask for help" sub-section within that of the v2.76f version of EP in which you said you were using. It is positioned at the bottom of the manual since the instructions begs everyone to read it thoroughly, it should be the last thing you read before going to ask for help. ;) Don't be embarrassed, you are actually in the norm rather then the exception. I'm really trying not to harass folks too much. Sometimes I can't help it. :'(

Edited March 25, 2008 by surfalot

[mark27uk3]

the section "XII. SUPPORT" of the manual. The "How to ask for help" sub-section within that of the v2.76f version of EP in which you said you were using. It is positioned at the bottom of the manual since the instructions begs everyone to read it thoroughly, it should be the last thing you read before going to ask for help. ;) Don't be embarrassed, you are actually in the norm rather then the exception. I'm really trying not to harass folks too much. Sometimes I can't help it. :'(

 

Hi Todd,

 

oh yeah theres the little blighter lol

 

Ok here is the info as requested:

 

1 ) PHP Version:

PHP Version : 5.2.5

MYSQL Version: 4.1.22

Server Type & version: Linux Server Apache/2.2.8 (Unix) mod_ssl/2.2.8 OpenSSL/0.9.8b mod_auth_passthrough/2.1 mod_bwlimited/1.4

 

2) Set EP_SHOW_EP_SETTINGS to true in the EP settings and then

Temp Dir:

/home/mydomain/public_html/temp/

Temp Dir is Writable

Magic Quotes is: off

Split files on: 300 records

Model Num Size: 200

Price with tax: false

Calc Precision: 2

Replace quotes: false

Field seperator: comma

Excel safe output: true

Preserve tab/cr/lf: false

Category depth: 7

Enable attributes: true

SEF Froogle URLS: false

More Pics: false

Unknown Pics: false

HTC: false

SPPC: false

Extra Fields: false

 

3) Copy of any error message you are receiving.

Warning: array_merge() [function.array-merge]: Argument #1 is not an array in /home/mydomain/public_html/admin/easypopulate.php on line 577

 

Warning: array_merge() [function.array-merge]: Argument #2 is not an array in /home/mydomain/public_html/admin/easypopulate.php on line 577

 

4) Post a link to a sample of any upload file you are trying.

No need to post a link to the upload file as I am trying to download >_<

 

Thanks

 

Mark

[surfalot]

Hi Todd,

 

oh yeah theres the little blighter lol

 

Ok here is the info as requested:

 

1 ) PHP Version:

 

 

2) Set EP_SHOW_EP_SETTINGS to true in the EP settings and then

 

 

3) Copy of any error message you are receiving.

 

 

4) Post a link to a sample of any upload file you are trying.

 

 

Thanks

 

Mark

What is happening is the code is hitting a product with at least one category entry in the products_to_categories table, in which the category's description is missing.

 

do you have any products that are not within a category?

 

or maybe a product that should be associated with a category, and the category's record is missing from the categories table?

 

if you add this just before that line, temporarily, and run again, it will tell you what product its barfing on...

echo "current product_id=" . $row['v_products_id'] . '<br>';
echo "missing category_id=" . $row['v_categories_id'];
exit;

[telefonica600]

what problem did you have with v2.76f or v2.76g ?

 

 

Every other version that I have loaded went up fine, appearance was correct, and permissions were good, but failed to execute any commands. The program would not read from nor write to the temp file. The screen would refresh appearing as it had completed something, but the refresh was the only thing completed.

There were no contents in the temp file. I could FTP to the temp file to manually add a file and it would not populate it.

I installed this version and was able to execute at least the upload portion.

I still can not get the downloads or quick links to work.

 

Steve

[surfalot]

Every other version that I have loaded went up fine, appearance was correct, and permissions were good, but failed to execute any commands. The program would not read from nor write to the temp file. The screen would refresh appearing as it had completed something, but the refresh was the only thing completed.

There were no contents in the temp file. I could FTP to the temp file to manually add a file and it would not populate it.

I installed this version and was able to execute at least the upload portion.

I still can not get the downloads or quick links to work.

 

Steve

I understand you are frustrated with "every other version". Can you be specific about the v2.76f or g ??

these are the only versions I will support, since at this point, "every other version" has significant issues that have been addressed in my latest versions.

[telefonica600]

I understand you are frustrated with "every other version". Can you be specific about the v2.76f or g ??

these are the only versions I will support, since at this point, "every other version" has significant issues that have been addressed in my latest versions.

 

 

I will have to reload them and try again to be specific to those two. I had the same issue with "ALL" the others that i tried.

 

And before someone decides to go newb busting on me!!(HAHAHA) My manual didn't include the "How to ask for help section.

So I downloaded another version and checked it.

As soon as i get through the DNS issue that I am having since they loaded my SSL Certificate (maybe coincidence) I will get that for you.

Edited March 26, 2008 by telefonica600

[telefonica600]

Easy Populate 2.76f-MS2 - Default Language : english(1)

 

PHP Version: 4.3.11 (Zend: 1.3.0)

Database: MySQL 4.1.22-max-log

 

 

Temp Dir:

/home/content/r/e/d/mydomainname/html/catalog/temp/

Temp Dir is Writable

Magic Quotes is: off

Split files on: 300 records

Model Num Size: 12

Price with tax: false

Calc Precision: 2

Replace quotes: true

Field seperator: comma

Excel safe output: true

Preserve tab/cr/lf: false

Category depth: 7

Enable attributes: true

SEF Froogle URLS: true

More Pics: false

Unknown Pics: false

HTC: true

SPPC: false

Extra Fields: true

 

This is what is listed at the top of the page. This downloaded from version 3.0.

 

Thanks,

Steve

Edited March 26, 2008 by telefonica600