OSCommerce resource usage probelm

[wiktor_1982]

Hello Everybody.

 

I have a pretty old OSCommerce installation at www.turbokolor.com/shop . The site has about 500 products (only about half are active) and 4000 customers. The shop receives aprox unique 1000 visits a day. The site is hosted on a shared host (lunarpages.com)

 

I have recently started getting high resource usage emails from my host. Since then I have followed the threads on optimizing the oscommerce instalaltion (removing queries, unsued modules, tax class upgrade), have implemented a thumbnailing mechanism (checked it is not generating files everytime a user visits the site), caching, etc and blocked tons of spambots by ip deny manager.

 

I have managed to get the Mem usage and mysql processes pretty much under control but my CPU time is flying high. Recently with only 15 users at one time on the site I was using 6% of the CPU time (allowed is 1%).

 

I realize that I might need to upgrade the server (do I really?) but at current usages my hosting company suggested a dedicated 6core, 6GB server which I think is ridiculous.

 

By following different advices and forums have also broken all my images (previously stored in one folder) to 50 different folders (not more than 100 in each folder), but that made absolutely no difference. The really weird thing is that my TOP CPU processes are all shown as JPG files (see below):

 

Stats for 05 Mar 2013:

---------------------------------

CPU Usage - %4.16

MEM Usage - %1.63

Number of MySQL procs (average) - 0.24

Top Process %CPU 6.20 httpd [turbokolor.myquizco.com] [/shop/images/48tkimg/ss13porcja026mini217x266.jpg]

Top Process %CPU 6.00 httpd [turbokolor.myquizco.com] [/shop/images/48tkimg/ss13porcja027mini217x266.jpg]

Top Process %CPU 5.00 httpd [turbokolor.myquizco.com] [/shop/images/banners/FW12/d3/BANNER_SKLEP_D03EE.jpg]

 

Could anyone throw some ideas why the CPU time can be so high and why is it pointing to images?

 

Thanks a lot in advance.

 

Wiktor

[wiktor_1982]

One more thing I forgot to mention is that all my products exist in at least 2 categories so that definietly makes for longer SQL query execution but sould not affect CPU time?

[MrPhil]

It is indeed odd that merely serving up image files would consume so much CPU. Are you doing something beyond merely throwing a file over the transom to the browser? Are you resizing an image on the server every time it's served, or anything like that? These are simply image files, and not kept in the database itself? How large are these image files -- you aren't sending out 4000 x 3000 pixel files and letting the browser resize them, are you?

 

I've heard complaints about LP being too quick on the draw to shut down users when their CPU usage exceeds 1%, but simply serving up an image should not consume all that much. As for SQL queries, they do consume CPU time (whether the server is localhost or on a separate DB server), but I don't know how LP counts that towards your CPU usage.

 

If you can't figure out what's going on, feel free to open a support ticket and ask for an explanation for how a simple image file can use so much CPU.

[wiktor_1982]

Hi and thanks for your answer.

 

I am working with their support but they are not being all that helpfull :/

 

To respond to your question, I am not serving big images. My original image (the one you see when you click on the product is about 100KB) and I made an extension in the html output class that reduces them to the right size (thumbnail of 10KB) and serves them to the user in the thumbnail size (and no its not doing that everytime the image is requested - creates it only once and then serves that instead of the original).

 

Do you think that with the amount of traffic/users/products I should be using so many resources?

 

Thanks,

 

Wiktor

[♥joli1811]

Hi Wicktor,

 

Few questions why can the images be found in x 2 places ??

 

http://turbokolor.com/shop/images/48tkimg/ss13porcja026mini217x266.jpg

 

http://turbokolor.myquizco.com/shop/images/48tkimg/ss13porcja026mini217x266.jpg

 

You seem to be running some sort of subdomain/s??

store.turbokolor.com

turbokolor.myquizco.com

 

Have you thought about hotlink protection??

 

Regards

Joli

[MrPhil]

Have you thought about hotlink protection?

 

Can you elaborate on that, John? Are you asking if they have hotlink protection, or are you suggesting that they add it? In the former case, that should be very little CPU load (in .htaccess). In the latter case, what will it do for Wiktor? Now, if images are being specially processed in .htaccess (reencoded, rescaled, whatever), that's a whole different thing.

[♥joli1811]

Hi,

@Mention

 

Well just do not understand the set up as I mentioned above if you follow the link http://turbokolor.myquizco.com/ he seems to be nearly a replica of store.turbokolor.com.

So was wondering if he is drawing images/ bandwidth from store.turbokolor.com

Regards

Joli

[MrPhil]

OK, myquizco.com is grabbing a lot of large image files from turbokolor.com. Serving them up could be fairly costly. Assuming myquizco.com is authorized to use these images (there's a Shop link in myquizco.com to turbokolor.com, so I would assume so), the question is whether something could be done to reduce CPU usage. First, are these images scaled down on the browser? I haven't looked at the slideshow code, but if they're 4000x3000 and you're scaling them down on the browser to 1000x750, that's a lot of wasted bandwidth (which costs CPU cycles even though it's "unlimited"). It would be even costlier to scale them down on the fly on the server side. Second, make sure your images are being cached by viewers' browsers, and there's nothing specified that keeps them from being cached.

 

A browser will need to fetch each image from the server, so it shouldn't make any difference whether you're specifying http://... on myquizco.com or just a local path on turbokolor.com. If myquizco.com is an add-on to your LP account (or vice-versa), it shouldn't matter, but if myquizco.com is hosted elsewhere, you're paying for that image file bandwidth via your turbokolor.com account. If you're paying for both domains, it shouldn't matter, unless bandwidth is much cheaper on one than the other.

[♥joli1811]

@@MrPhil

 

Well Mr Phil learnt a bit about CPU usage today I always thought it would be directly linked to bandwidth usage but after a bit of reading I see it is not necessarily so.

 

Here the link to the slideshow seems they must be scaled on the fly if I understand it correctly.

 

http://buildinternet.com/2011/07/supersized-3-2-fullscreen-jquery-slideshow/

 

PS:

 

Have not seen a way to read CPU usage from a cpanel or WHM account so makes a bit hard to debug suppose only the server can see??

 

regards

Joli

[MrPhil]

I'm at work right now and can't dive into this slideshow's code. By "scaled on the fly" do you mean on the server or on the browser? If on the server, it of course means driving up the CPU bill there. If on the browser, it means serving potentially oversized image files, which still costs CPU cycles on the server. Either way, this slideshow may be making the difference between being allowed by the host and exceeding limits. Short of dropping the slideshow or cutting the number of slides, one thing Wiktor could do is make sure that his image files are scaled down to no larger than the largest size they'll be shown at on the browser, so little or no scaling needs to be done on either side, and bandwidth use is minimized.

 

If the slideshow generates thumbnail images on the fly, then it's very poorly designed. It should create (or accept) pre-generated thumbnails so that the conversion is done only once, at most. Can you tell what it's doing?

 

Lunarpages is notorious for shutting down sites which exceed some CPU load limit, yet do not provide any way for their customers to monitor their CPU usage.

[♥joli1811]

@@MrPhil

 

Hi had another look there are a load of images being called by the supersized jquery

 

http://turbokolor.myquizco.com/images/promo112012/laufer/stronaTKglownaRELAXD03.jpg

 

3,840px × 2,400px (scaled to 993px × 621px)

 

Do not understand enough to know what it is doing but do understand enough to (be dangerous :D )

see this jquery as a likely suspect

 

https://github.com/buildinternet/supersized

 

http://my.opera.com/community/forums/topic.dml?id=1555452

 

Although opera just thought it was interesting that they mentioned high CPU issue with a fix.

 

regards

John

[MrPhil]

So the server is pumping out almost 16 times as many bytes as needed? That doesn't sound very good to me. If the images will be displayed no larger than 993x621, I would reduce them on the server (using GIMP, Photoshop, etc. on a PC) to that size. Keep the originals around somewhere safe so you can use them again, but day-to-day usage should have little or no scaling needed. Eliminate any server-side on-the-fly scaling and don't transmit more bytes than you need to. If this slideshow package can accept pre-built thumbnail images, rather than creating them on the fly on the server, that would be a good thing to do just once.

[wiktor_1982]

Hi Guys,

 

Thank you for all the replies. I was on vacation without access to the net for a while but now read all your points and here are the responses:

 

@@joli1811

all the locations. Such as turbokolor.myquizco.com/shop, store.turbokolor.com, turbokolor.com/shop are just subdomians. All the images files are in one location. And yes, hotlink protection is enabled

 

@@MrPhil. The slideshow on turbokolor.com is scaling the images on the browser side. And yes the served images are very big (to allow for different resolutions). Was so caught up in thinking it must be oscommerce that didn't consider the slideshow as a likely culprit. The images are however cached. Cacheing set to 1 month in htaccess

 

Is it possible that 3 slides (even thath big) would couse such high resource usage?

 

I read the link about cpu usage in opera but thats client side and should not affect server cpu usage short of serving those 3 big slides. But then again why wouldn't it show up in my topprocesses (my top processes are still the small image files from oscommerce)?

 

Best,

Wiktor

[wiktor_1982]

One more question. No one has really commented on the size of the shop. Am I correct in understanding this is a pretty small setup?

 

Best,W

[♥joli1811]

Hi Wiktor,

 

Got me beat the shop looks fine to me not a big setup think probably rc2.

 

Seems you have already done a lot of work already optimizing database queries.

 

Did you do the one about optimizing the category queries ??

 

Not many ideas nice thread here

 

http://www.freehostia.com/blog/webhosting/reduce-cpu-usage-advanced-tactics.html

 

But guess you have done most of that as well, 1000 hits a day is good but not extreme.

 

But maybe some of the more experienced guys may have an idea or two :) .

Regards

Joli

[wiktor_1982]

Hi,

 

Honestly I don't remember now about the categories query. Which one is it? My thinking though is that if my MEM usage is well within limits than it is not due to MySQL queries.

 

About the link you posted. I checked my Bandwidth since August (last major update to the website) and since August we do have a significant increase in Bandwidth (thats when the slideshow first started, and we started using larger images in oscommerce) and since then the bandwidth used is 2-3 times bigger than before. Now in March and April after the changes the bandwidth is down again (after implementing caching and resizing images before serving them).

 

Also my top KBs byt URL is the 3 large files of the slideshow - I just had no idea that serving large images can be so CPU intensive :/

 

I'll see how I can optimize this slideshow jquery plugin - though not sure I can do it :/

 

Keep posting ideas! Thanks

 

Best,

W

W

[wiktor_1982]

Also I am have just now installed the Security Pro addon but can;t get it to work. Not sure why. When I follow the authors instructions on testing it I still get the bad characters in my query string :/ Any ideas?

 

W

[wiktor_1982]

never mind :) was using the wrong version....

[wiktor_1982]

Anyways. Its working now. But what pointed me in this direction was the fact that recently we received a couple of orders where all the data other than the shipping address was empty. The orders were all authorized by the external payment system thru a bank account of the customer but it seems really weird that such order could even be placed. Has anyone seen anything like that before?

 

 

Best,

 

W