surfalot Posted March 20, 2008 Posted March 20, 2008 What's the most number of lines anyone's been able to use Easy Populate to import at once? I think my record is about 18,000... which, given that I have a 50,000 line .csv to import, doesn't help. I've been trying to get some community support on that for months to no avail. But, anyway. I'm curious to just -know- if anyone's been able to clear that 50,000 plateau, or something significantly larger? that depends on your host and what you can get away with. So you can't split that into 3 and process 3 batches? if you host will allow you to override PHP script run limit with the set_time_limit setting, the sky is the limit. Quote
winterion Posted March 20, 2008 Posted March 20, 2008 (edited) that depends on your host and what you can get away with. So you can't split that into 3 and process 3 batches?if you host will allow you to override PHP script run limit with the set_time_limit setting, the sky is the limit. My host (Bluehost) permits you to put in your own php.ini file, so I've done so. (I wasn't sure where to put it, so I put a copy in public_html, a copy in my osCommerce root dir, and a copy in the admin directory. They are all identical files.) I've never heard of the set_time_limit variable. I've adjusted the following in php.ini: max_execution_time = 7200 ; Maximum execution time of each script, in seconds (30) max_input_time = 7200 ; Maximum amount of time each script may spend parsing request data (60) memory_limit = 256M ; Maximum amount of memory a script may consume (32MB) And, I've adjusted the following in easypopulate.php: (the server does not run in safe mode) set_time_limit(0); define ('EP_SPLIT_MAX_RECORDS', 10000); I've even edited the function/sessions.php file: $SESS_LIFE = 28800; So, honest to goodness, I'm stumped. Edit: Forgot to add this: EP vers: 2.76f-MS2 r1 Temp Dir: /home/gameparl/public_html/catalog/temp/ Temp Dir is Writable Magic Quotes is: off register_globals is: off Split files on: 10000 records Model Num Size: 24 Price with tax: false Calc Precision: 2 Replace quotes: false Field seperator: comma Excel safe output: true Preserve tab/cr/lf: false Category depth: 24 Enable attributes: false SEF Froogle URLS: false More Pics: false Unknown Pics: false HTC: false SPPC: false Extra Fields: false Edited March 20, 2008 by winterion Quote
surfalot Posted March 20, 2008 Posted March 20, 2008 My host (Bluehost) permits you to put in your own php.ini file, so I've done so.(I wasn't sure where to put it, so I put a copy in public_html, a copy in my osCommerce root dir, and a copy in the admin directory. They are all identical files.) I've never heard of the set_time_limit variable. I've adjusted the following in php.ini: max_execution_time = 7200 ; Maximum execution time of each script, in seconds (30) max_input_time = 7200 ; Maximum amount of time each script may spend parsing request data (60) memory_limit = 256M ; Maximum amount of memory a script may consume (32MB) And, I've adjusted the following in easypopulate.php: (the server does not run in safe mode) set_time_limit(0); define ('EP_SPLIT_MAX_RECORDS', 10000); I've even edited the function/sessions.php file: $SESS_LIFE = 28800; So, honest to goodness, I'm stumped. Edit: Forgot to add this: EP vers: 2.76f-MS2 r1 Temp Dir: /home/gameparl/public_html/catalog/temp/ Temp Dir is Writable Magic Quotes is: off register_globals is: off Split files on: 10000 records Model Num Size: 24 Price with tax: false Calc Precision: 2 Replace quotes: false Field seperator: comma Excel safe output: true Preserve tab/cr/lf: false Category depth: 24 Enable attributes: false SEF Froogle URLS: false More Pics: false Unknown Pics: false HTC: false SPPC: false Extra Fields: false your best bet is to split the import file if you can't force the server to process the whole file. how long does it take to process 18,000 records? does it stop on the same one each time? Quote
winterion Posted March 21, 2008 Posted March 21, 2008 your best bet is to split the import file if you can't force the server to process the whole file. how long does it take to process 18,000 records? does it stop on the same one each time? The problem we have with splitting the import file is logistical.. I have found by splitting it into 10 files, 5000 or so each, we can reliably upload the whole thing. But, that means, every half-hour I need whoever is working the store to perform the next step. The hope was that whoever is working, once a week, can simply start this process up, and that it would go, start to finish, on its own. :blush: Or, is there an automated method in which EP can perform one .csv file in a series of uploads? (would you need Row 1 repeated, say, every 5000 lines? Could do that with an Excel macro, for example.) Quote
surfalot Posted March 21, 2008 Posted March 21, 2008 The problem we have with splitting the import file is logistical.. I have found by splitting it into 10 files, 5000 or so each, we can reliably upload the whole thing. But, that means, every half-hour I need whoever is working the store to perform the next step. The hope was that whoever is working, once a week, can simply start this process up, and that it would go, start to finish, on its own. :blush: Or, is there an automated method in which EP can perform one .csv file in a series of uploads? (would you need Row 1 repeated, say, every 5000 lines? Could do that with an Excel macro, for example.) so why not use the split function of EP? and why not answer the couple questions in the last post? Quote
winterion Posted March 21, 2008 Posted March 21, 2008 (edited) so why not use the split function of EP? and why not answer the couple questions in the last post? Oh, I'm really sorry, big gaffe on my part. :-" Umm, the 18,000 or so takes somewhere around 15 minutes? Maybe less.. I could time it exactly. I'm totally new to EP, so I'm not exactly familiar how the 'split' function works. I can try and look it up, try to learn, won't beg you guys to hold my hand through that. Should it be expected that a 50,000 line .csv file would have to use the split function? As for the same one each time? No, it doesn't.. different line each time. So I don't think the stop is related to the .csv file itself, like, a bad line in it. Edit: Okay, my idiocy knows no bounds. The split thing was really easy. But, those five files it created (since my split was at 10k items).. any way to command EP to import those one at a time, all five, or do I have to keep coming back to the PC to do them one at a time? (Sort of a queue thing..) Edited March 21, 2008 by winterion Quote
surfalot Posted March 21, 2008 Posted March 21, 2008 (edited) man, this board software is going to the dogs!! I have to clear out the emot icons before posting a reply now or everything else is cut off! :angry: Oh, I'm really sorry, big gaffe on my part. Umm, the 18,000 or so takes somewhere around 15 minutes? Maybe less.. I could time it exactly. I'm totally new to EP, so I'm not exactly familiar how the 'split' function works. I can try and look it up, try to learn, won't beg you guys to hold my hand through that. Should it be expected that a 50,000 line .csv file would have to use the split function? As for the same one each time? No, it doesn't.. different line each time. So I don't think the stop is related to the .csv file itself, like, a bad line in it. Edit: Okay, my idiocy knows no bounds. The split thing was really easy. But, those five files it created (since my split was at 10k items).. any way to command EP to import those one at a time, all five, or do I have to keep coming back to the PC to do them one at a time? (Sort of a queue thing..) OK once more... so it would have been a good idea to have read through the manual like the install instructions begged you to, ey? >_< There is a small section about large files. Its not expected to have to split a large file, it depends on what your host will let you get away with. I'd say it is at your host. yes you have to run each one manually. If anyone has a suggestion for that, I'm all ears. I haven't put much thought into it. Edited March 21, 2008 by surfalot Quote
veni Posted March 22, 2008 Posted March 22, 2008 Hello guys, I have a site that is very simple example, no prices are listed. I wish to remove some of the column headers in Easy Populate, these items are : v_products_name_2 v_products_url_2 v_products_description_2 v_products_name_3 v_products_url_3 v_products_description_3 v_products_price v_products_quantity v_products_weight v_date_avail v_date_added v_categories_image_1 v_categories_name_2 v_categories_image_2 v_categories_name_3 v_categories_image_3 v_categories_name_4 v_categories_image_4 v_categories_name_5 v_categories_image_5 v_categories_name_6 v_categories_image_6 v_categories_name_7 v_categories_image_7 v_tax_class_title Is is possible to remove these Items as I don't use them in my site at all? Quote
surfalot Posted March 22, 2008 Posted March 22, 2008 Hello guys, I have a site that is very simple example, no prices are listed. I wish to remove some of the column headers in Easy Populate, these items are : Is is possible to remove these Items as I don't use them in my site at all? yes, if you don't need to change a bit of data, you don't need to include the column. exception would be if you are using some earlier versions that if you include any of the product name, description or URL, you must include all three or you could loose data. always backup, give it a try. Quote
roberovsky Posted March 23, 2008 Posted March 23, 2008 has anyone make easy populate make work for the downloading files Quote
telefonica600 Posted March 23, 2008 Posted March 23, 2008 (edited) Ok, so I have been using easypopulate to upload the 2500 items in my store. The contrib is GREAT! I want to do a mass price change. I can edit my .CSV file to include the price change, but I can not get it to upload. Will easy populate do an update of existing items? The only way that I have been able to change it is to delete the category and reload it. The problem there is that I need to reload all of the pictures after I delete the category. It turns into a LOT of FTP time. Any suggestions? Steve www.reddogenterprises.net Edited March 23, 2008 by telefonica600 Quote
surfalot Posted March 24, 2008 Posted March 24, 2008 has anyone make easy populate make work for the downloading files someone uploaded a version for a specific download contribution, not the standard download product built in. Other then that, not yet. Quote
surfalot Posted March 24, 2008 Posted March 24, 2008 Ok, so I have been using easypopulate to upload the 2500 items in my store.The contrib is GREAT! I want to do a mass price change. I can edit my .CSV file to include the price change, but I can not get it to upload. Will easy populate do an update of existing items? The only way that I have been able to change it is to delete the category and reload it. The problem there is that I need to reload all of the pictures after I delete the category. It turns into a LOT of FTP time. Any suggestions? Steve www.reddogenterprises.net EP vers? Quote
telefonica600 Posted March 25, 2008 Posted March 25, 2008 EP vers? easypopulate.php,v 3.00 (Based on 2.78, 2.80 and 2.81b) This is the only one that I have successfully been able to get to work. I am not sure why, but I can only import from the temp directory. I can not use this to export or write to the temp directory. I have gotten farther with this version than any other version. I have successfully loaded almost 3000 products at once. Thanks for the help! I appreciate the time. Quote
surfalot Posted March 25, 2008 Posted March 25, 2008 easypopulate.php,v 3.00 (Based on 2.78, 2.80 and 2.81b) This is the only one that I have successfully been able to get to work. I am not sure why, but I can only import from the temp directory. I can not use this to export or write to the temp directory. I have gotten farther with this version than any other version. I have successfully loaded almost 3000 products at once. Thanks for the help! I appreciate the time. what problem did you have with v2.76f or v2.76g ? Quote
mark27uk3 Posted March 25, 2008 Posted March 25, 2008 Hi Guys, I hope someone can point me in the right direction here as I have read and read and read and googled and googled (you get the picture) Either I am being extremely dense or just plain blind. I am pretty sure that my problem is a php5 and can find others with the same problem but no resolution. I am using EP 2.76f Warning: array_merge() [function.array-merge]: Argument #1 is not an array in /home/mydomain/public_html/admin/easypopulate.php on line 577 Warning: array_merge() [function.array-merge]: Argument #2 is not an array in /home/mydomain/public_html/admin/easypopulate.php on line 577 This is line 577 $row2 = array_merge($row2, tep_db_fetch_array($result2)); Hoping that someone else has had this issue and resolved it. Thanks Mark Quote Lifes a bitch, then you marry one, then you die!
surfalot Posted March 25, 2008 Posted March 25, 2008 Hi Guys, I hope someone can point me in the right direction here as I have read and read and read and googled and googled (you get the picture) Either I am being extremely dense or just plain blind. I am pretty sure that my problem is a php5 and can find others with the same problem but no resolution. I am using EP 2.76f This is line 577 $row2 = array_merge($row2, tep_db_fetch_array($result2)); Hoping that someone else has had this issue and resolved it. Thanks Mark you have certainly done the prereq searching. how about offering up the info asked for in the "how to ask for help" section of the docs. Quote
mark27uk3 Posted March 25, 2008 Posted March 25, 2008 you have certainly done the prereq searching. how about offering up the info asked for in the "how to ask for help" section of the docs. ok now I am being blind which part of the docs would that be (now I feel :blush: ) Mark Quote Lifes a bitch, then you marry one, then you die!
surfalot Posted March 25, 2008 Posted March 25, 2008 (edited) ok now I am being blind which part of the docs would that be (now I feel :blush: ) Mark the section "XII. SUPPORT" of the manual. The "How to ask for help" sub-section within that of the v2.76f version of EP in which you said you were using. It is positioned at the bottom of the manual since the instructions begs everyone to read it thoroughly, it should be the last thing you read before going to ask for help. ;) Don't be embarrassed, you are actually in the norm rather then the exception. I'm really trying not to harass folks too much. Sometimes I can't help it. :'( Edited March 25, 2008 by surfalot Quote
mark27uk3 Posted March 25, 2008 Posted March 25, 2008 the section "XII. SUPPORT" of the manual. The "How to ask for help" sub-section within that of the v2.76f version of EP in which you said you were using. It is positioned at the bottom of the manual since the instructions begs everyone to read it thoroughly, it should be the last thing you read before going to ask for help. ;) Don't be embarrassed, you are actually in the norm rather then the exception. I'm really trying not to harass folks too much. Sometimes I can't help it. :'( Hi Todd, oh yeah theres the little blighter lol Ok here is the info as requested: 1 ) PHP Version: PHP Version : 5.2.5 MYSQL Version: 4.1.22 Server Type & version: Linux Server Apache/2.2.8 (Unix) mod_ssl/2.2.8 OpenSSL/0.9.8b mod_auth_passthrough/2.1 mod_bwlimited/1.4 2) Set EP_SHOW_EP_SETTINGS to true in the EP settings and then Temp Dir:/home/mydomain/public_html/temp/ Temp Dir is Writable Magic Quotes is: off Split files on: 300 records Model Num Size: 200 Price with tax: false Calc Precision: 2 Replace quotes: false Field seperator: comma Excel safe output: true Preserve tab/cr/lf: false Category depth: 7 Enable attributes: true SEF Froogle URLS: false More Pics: false Unknown Pics: false HTC: false SPPC: false Extra Fields: false 3) Copy of any error message you are receiving. Warning: array_merge() [function.array-merge]: Argument #1 is not an array in /home/mydomain/public_html/admin/easypopulate.php on line 577 Warning: array_merge() [function.array-merge]: Argument #2 is not an array in /home/mydomain/public_html/admin/easypopulate.php on line 577 4) Post a link to a sample of any upload file you are trying. No need to post a link to the upload file as I am trying to download >_< Thanks Mark Quote Lifes a bitch, then you marry one, then you die!
surfalot Posted March 25, 2008 Posted March 25, 2008 Hi Todd, oh yeah theres the little blighter lol Ok here is the info as requested: 1 ) PHP Version: 2) Set EP_SHOW_EP_SETTINGS to true in the EP settings and then 3) Copy of any error message you are receiving. 4) Post a link to a sample of any upload file you are trying. Thanks Mark What is happening is the code is hitting a product with at least one category entry in the products_to_categories table, in which the category's description is missing. do you have any products that are not within a category? or maybe a product that should be associated with a category, and the category's record is missing from the categories table? if you add this just before that line, temporarily, and run again, it will tell you what product its barfing on... echo "current product_id=" . $row['v_products_id'] . '<br>'; echo "missing category_id=" . $row['v_categories_id']; exit; Quote
telefonica600 Posted March 26, 2008 Posted March 26, 2008 what problem did you have with v2.76f or v2.76g ? Every other version that I have loaded went up fine, appearance was correct, and permissions were good, but failed to execute any commands. The program would not read from nor write to the temp file. The screen would refresh appearing as it had completed something, but the refresh was the only thing completed. There were no contents in the temp file. I could FTP to the temp file to manually add a file and it would not populate it. I installed this version and was able to execute at least the upload portion. I still can not get the downloads or quick links to work. Steve Quote
surfalot Posted March 26, 2008 Posted March 26, 2008 Every other version that I have loaded went up fine, appearance was correct, and permissions were good, but failed to execute any commands. The program would not read from nor write to the temp file. The screen would refresh appearing as it had completed something, but the refresh was the only thing completed.There were no contents in the temp file. I could FTP to the temp file to manually add a file and it would not populate it. I installed this version and was able to execute at least the upload portion. I still can not get the downloads or quick links to work. Steve I understand you are frustrated with "every other version". Can you be specific about the v2.76f or g ?? these are the only versions I will support, since at this point, "every other version" has significant issues that have been addressed in my latest versions. Quote
telefonica600 Posted March 26, 2008 Posted March 26, 2008 (edited) I understand you are frustrated with "every other version". Can you be specific about the v2.76f or g ??these are the only versions I will support, since at this point, "every other version" has significant issues that have been addressed in my latest versions. I will have to reload them and try again to be specific to those two. I had the same issue with "ALL" the others that i tried. And before someone decides to go newb busting on me!!(HAHAHA) My manual didn't include the "How to ask for help section. So I downloaded another version and checked it. As soon as i get through the DNS issue that I am having since they loaded my SSL Certificate (maybe coincidence) I will get that for you. Edited March 26, 2008 by telefonica600 Quote
telefonica600 Posted March 26, 2008 Posted March 26, 2008 (edited) Easy Populate 2.76f-MS2 - Default Language : english(1) PHP Version: 4.3.11 (Zend: 1.3.0) Database: MySQL 4.1.22-max-log Temp Dir: /home/content/r/e/d/mydomainname/html/catalog/temp/ Temp Dir is Writable Magic Quotes is: off Split files on: 300 records Model Num Size: 12 Price with tax: false Calc Precision: 2 Replace quotes: true Field seperator: comma Excel safe output: true Preserve tab/cr/lf: false Category depth: 7 Enable attributes: true SEF Froogle URLS: true More Pics: false Unknown Pics: false HTC: true SPPC: false Extra Fields: true This is what is listed at the top of the page. This downloaded from version 3.0. Thanks, Steve Edited March 26, 2008 by telefonica600 Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.