sludgepit Posted March 5, 2003 Share Posted March 5, 2003 Hi! I'm trying to use EasyPopulate (2.4b7) to upload products into osCommerce. The thing is that the data source is delivered as a whole bunch of seperate TAB delimted text files (ie pricing in a seperate file to product details etc.). When uploading a text file using EasyPopulate, does the file you are trying to import have to have ALL fields present (even if they are blank) ? What I'm hoping is that I could do something like this: Modify the product details text file, and include blank columns for fields which are located in another file (eg. pricing), then upload that file. Then, modify the pricing file, and include columns for each field (empty), then upload THAT file. Is this a valid way of working, or do I have to somehow merge the individual files such that ALL the info for each product is populated in the one TAB delimted text file? Thanks for any help in advance... Quote Cheers Robert Link to comment Share on other sites More sharing options...
wasson65 Posted March 7, 2003 Share Posted March 7, 2003 Hi! I'm trying to use EasyPopulate (2.4b7) to upload products into osCommerce. The thing is that the data source is delivered as a whole bunch of seperate TAB delimted text files (ie pricing in a seperate file to product details etc.). When uploading a text file using EasyPopulate, does the file you are trying to import have to have ALL fields present (even if they are blank) ? What I'm hoping is that I could do something like this: Modify the product details text file, and include blank columns for fields which are located in another file (eg. pricing), then upload that file. Then, modify the pricing file, and include columns for each field (empty), then upload THAT file. Is this a valid way of working, or do I have to somehow merge the individual files such that ALL the info for each product is populated in the one TAB delimted text file? Thanks for any help in advance... I think this got answered in another thread. Were you able to get it working the way you wanted? Tim Quote Link to comment Share on other sites More sharing options...
sludgepit Posted March 11, 2003 Author Share Posted March 11, 2003 Well...I'm almost there...but I've been learning a lot about this stuff tho... One of the problems I have is that I have to "massage" the data before I even try to import it. ie In the pricing file I have to take a specified price field and calculate their sell price (source price*1.25 to add 25% for example). I have been messing around with a product called "ParseRat" which can suck in a TAB delimited file, modify field names etc, perform calcs, and then export to another file relatively easy. This program is pretty rubbishy, though, I'm getting various errors making it hard to process these files. (PS you're not aware of any easy to use utilities to do this sorta thing are you???) I now have the split function working nicely...thank you, and I can import my files okay, my only drama is smoothing the files out before import now... Thanks for your help Quote Cheers Robert Link to comment Share on other sites More sharing options...
wasson65 Posted March 11, 2003 Share Posted March 11, 2003 Well...I'm almost there...but I've been learning a lot about this stuff tho... One of the problems I have is that I have to "massage" the data before I even try to import it. ie In the pricing file I have to take a specified price field and calculate their sell price (source price*1.25 to add 25% for example). I have been messing around with a product called "ParseRat" which can suck in a TAB delimited file, modify field names etc, perform calcs, and then export to another file relatively easy. This program is pretty rubbishy, though, I'm getting various errors making it hard to process these files. (PS you're not aware of any easy to use utilities to do this sorta thing are you???) I now have the split function working nicely...thank you, and I can import my files okay, my only drama is smoothing the files out before import now... Thanks for your help If you're just doing a simple percentage calc, why not mod EP to do it for you? Just look for the insert queries where it has product_price, and multiply there. Tim Quote Link to comment Share on other sites More sharing options...
sludgepit Posted March 12, 2003 Author Share Posted March 12, 2003 I'd rather not mess with your code - I'm not a programmer 8-) Besides, There's a whole bunch of other things I have to do (like strip out stupid characters and replace them - the supplier gives us text files with an OCTAL 375 as a logical end of line in their descriptions, I want to change this to a HTML <BR>). I've gone back to using "gawk" for this stuff, although I'm having trouble trying to figure out how to substitute <BR> for 0x375... Thanks for your help Quote Cheers Robert Link to comment Share on other sites More sharing options...
sludgepit Posted March 12, 2003 Author Share Posted March 12, 2003 BTW - The "Split EP File" function does not work if the file is bigger than the "upload_max_filesize" limit specified in /etc/php.ini If this occurs, EasyPopulate just returns with no error message. I turned on PHP logging to syslog, and restarted Apache to find the error in syslog. FYI Quote Cheers Robert Link to comment Share on other sites More sharing options...
sludgepit Posted March 12, 2003 Author Share Posted March 12, 2003 ...another thing I have noticed is When I'm importing split files the CPU usage for mysqld zooms up to around 95%, and the import takes AGES. These split files are large (3000 lines each, except for the last one), but I am uploading locally across a LAN, so I didn;t think the size would be such an issue. Do you know of any specific tweaks for mysql which might improve the speed of imports using EasyPopulate ?? Thanks again for your help Quote Cheers Robert Link to comment Share on other sites More sharing options...
wasson65 Posted March 14, 2003 Share Posted March 14, 2003 ...another thing I have noticed is When I'm importing split files the CPU usage for mysqld zooms up to around 95%, and the import takes AGES. These split files are large (3000 lines each, except for the last one), but I am uploading locally across a LAN, so I didn;t think the size would be such an issue. Do you know of any specific tweaks for mysql which might improve the speed of imports using EasyPopulate ?? Thanks again for your help EasyPopulate does a lot of queries. Each row it checks to see if the categories exist, etc. There might be some optimizations in the code that would reduce the amount of database work it has to do, and I'd guess that's where your biggest work is. If you've added bunches of indexes to your products tables, then you might kill the indexes, do your imports, and remake your indexes. Indexes can slow down updates and inserts. Tim Quote Link to comment Share on other sites More sharing options...
Guest Posted June 23, 2005 Share Posted June 23, 2005 ...another thing I have noticed isWhen I'm importing split files the CPU usage for mysqld zooms up to around 95%, and the import takes AGES. These split files are large (3000 lines each, except for the last one), but I am uploading locally across a LAN, so I didn;t think the size would be such an issue. Do you know of any specific tweaks for mysql which might improve the speed of imports using EasyPopulate ?? Thanks again for your help <{POST_SNAPBACK}> Im not sure if this will help you, but I think its worth it to give it a go: 23 Jun 2005 - Easypopulate Attributes Optimization http://www.oscommerce.com/community/contributions,500 I had a database of 1600 products with over 250,000 different attributes, and originally it would take up to 18 minutes (if the script didnt timeout first) Now it takes about 20 seconds. ---sweet (Times may vary for you based on your server) All this optimization does is first load the entire attributes and options tables into an arrays. Then while its going product by product, it no longer makes a seperate mysql query calls, it just grabs the data from the existing array. Will work on EP v2.4+ Quote Link to comment Share on other sites More sharing options...
New 
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.