EasyPopulate problem

[winterion]

Okay. I'm sure this is a problem people have encountered before, and I hate to annoy you with things, but I swear that I've gone through the documentation a hundred times, tried all the basic solutions, and still, to no avail. :blush:

 

I have a 50,000 item .csv file I want to upload into my database on a weekly process so that the proper quantities of items display on our site, and new items get added.

 

Now, this .csv file is easily enough created. And, for demonstration purposes, here's an example pair of lines.

 

v_products_model,v_categories_name_1,v_products_name_1,v_products_price,v_produc

ts_quantity,EOREOR

NOV02 2034,COMIC,Mekanix #4 (of 6),2.99,5,EOREOR

 

So, we've got the model number, the item category, the description (which is a cleverly concat'ed pair of lines in our database), price, quantity, and the end of row thingy.

So far, so good, right? Now, add 49998 more lines of the second line, with different stuff in it, obviously.

 

Here's the problem. We can upload it to the temp directory just fine. We can hit the import from temp dir, it starts processing, and in fact, it processes every entry it gets to just fine, accurately filling our SQL database with precious products!

 

Now, the trick is, it stops. It just.. stops. No error message, but somewhere between 12,000 and 18,000-ish items in, (usually somewhere around C to E) it just ends.

 

I'm not sure WHAT it is, but I suspect it's one of the following:

-osCommerce admin session timeout

-php.ini file settings

-easypopulate.php file settings

-dark sorcery

 

Now, here's the thing. I've read over all these other posts, I've tried (no exaggeration here) about 100 different settings for the php.ini and easypopulate.php files. I've put copies of php.ini in my public_html, catalog, and admin folders.

 

Anyway, the critical settings I've tried:

 

php.ini

max_execution_time = xxxx ; Maximum execution time of each script, in seconds (tried between 30 and 9000)

max_input_time = xxxx ; Maximum amount of time each script may spend parsing request data (tried between 30 and 9000)

memory_limit = xxxM ; Maximum amount of memory a script may consume (tried between 8M and 512M)

 

easypopulate.php

set_time_limit(xxx); have tried this commented and uncommented between 120 and 28800

define ('EP_SPLIT_MAX_RECORDS', xxx); have tried this commented and uncommented between 300 and 70000

$ep_separator = "\t"; now, interestingly, this setting seems to work for the comma-delimited .csv? or does this mean something else...

define ('EP_MAX_CATEGORIES', 24); we have about twenty categories, so this was to allow for a bit of expansion, no problems here

 

Now, this is what happens:

 

Easy Populate 2.76f-MS2 r1 - Default Language : english(1)

 

Filename: finalbook.csv

| NOV02 2034 | COMIC | Mekanix #4 | 2.99 | 5 Updated

| NOV02 2035 | COMIC | Mekanix #5 | 2.99 | 5 Updated

etc...

 

now, this list seems to keep building every time I tap the down key or page down or scroll.. while the browser continues to show a loading bar.

..for a few minutes.

 

Eventually, the status bar reads Stopped, and the last line usually looks something like

| NOV02 8035 | COMIC | Abbe

(where it's just cut off at some random point in the line).

 

Now, I'm so boggled.

I can't seem to figure out -what- causes it to stop.. be it a session timeout from osCommerce, or EasyPopulate, or the length of the massive SQL command that a 50,000 item .csv file creates, or what.. but I'm being told from a lot of people that there SHOULD be no problem with this.

But, I've been agonizing over this (amongst other problems.. whoever said this was easy is either lucky or a great liar!) for well over three months now. This was a project I started with the hopes of having it ready around Christmas, and now Easter is around the corner. Frankly, I feel sort of inept, so I've been throwing my various problems at the mercy of the community at large (some threads going un-noticed and un-replied, others full of the same suggestions I've seen elsewhere, to no avail.)

 

So, please, all I can do is ask.. help? >_<

Edited March 19, 2008 by winterion