PDA

View Full Version : Enormous memory leak with image sequences?



DiscoBurgess
07-23-2008, 04:29 PM
Having just read through the BSG VFX guys Emmy nomination submission, I decided to try out their image-sequences-on-hypervoxels-sprites technique, and I'm seeing massive memory usage in Lightwave, to the point that it locks up and won't finish the render, because it's used all my memory. Watching task manager as the render progresses, I can see it increases memory usage by tens of megabytes after every frame featuring the image sequences. The previous 160 frames are fine, but 20-25 frames of four image sequences, and the memory usage just carries on growing.

I took two of the image sequences out, and sure enough, memory goes up at about half the rate.

Anyone else seeing this? It's a bit of a serious problem. Perhaps it doesn't happen in the 64 bit version, or the BSG guys have much more than than my 2Gb RAM. Or both.

BeeVee
07-23-2008, 05:19 PM
What sort of images are you using? Avoid JPGs where possible and use 256-colour PNGs where you can. This will cut down on memory usage considerably. Optimized image use tutorial on LightWiki (http://www.lightwiki.com/Optimised_image_use).

B

Lightwolf
07-23-2008, 05:21 PM
It could be that it uses that much memory. The BSG VFX team is afaik using 64-bit LW exclusively (at least for Layout and rendering).

Cheers,
Mike

DiscoBurgess
07-24-2008, 02:36 AM
I'm using 24-bit TGAs and Quicktimes, but good point, I can convert them all to 8-bit PNGs no problem.

I've no problem with it using a lot of memory, I'm more concerned with the fact that memory usage steadily increases, like it's loading in the new frames, and not ditching the old ones. In the scene, the number of individual image sequence frames that need to be rendered per fame goes up, then back down. And yet memory just goes up with every frame that has any hint of an image sequence in.

I'll try the 8-bit solution, but it seems like there's something that needs fixing in Lightwave itself. In 25 frames, I'm a GB of memory down. Even with 8GB, you're still looking at less than 200 frames rendering at a time (assuming half a gig is being used for your scene and OS). It just seems, well, really bad.

DiscoBurgess
07-24-2008, 02:50 AM
Ack, can't edit my post.

I tried it without the Quicktimes, and the two 24-bit sequences themselves chewed up plenty of memory. I could get about 40 frames done before it crapped out.

But yeah, I'll try 8bit PNGs. Thanks for the info/suggestions, guys. Is this a bug, or just an unavoidable side-effect?

Nangleator
07-24-2008, 06:58 AM
Set up a screamernet render. Just break your scene into smaller render chunks. Frames 1-5, then frames 6-10, etc.

Perhaps memory will be cleared between scene loads.

(:hey:)

DiscoBurgess
07-25-2008, 02:35 AM
Okay, well, I switched to 8bit images, and it helped a lot. It still managed to build up 800Mb extra of used memory by the end of the render. So thanks for the advice, BeeVee, it definitely helped.

However, if I could be permitted a moan (sorry):

This still seems to me like a stupid memory leak. To take the situation above, if I quit lightwave, restart it, and render out any given frame, it manages it without the 800Mb overhead generated from the rendering (just taking up the usual 500Mb for the scene). So, clearly, that memory build-up is simply not needed. I'm going to take a guess here, and suggest that each frame isn't discarded from memory on subsequent renders, and new frames are loaded each time. The old frames stay in memory until I quit Lightwave.

Come on, that's rubbish. I admit, I've never written a raytracing package, but if I'm right, this should be an easy one to fix. I can't see how the images I was using originally could account for that much memory use, unless they weren't being tidied from memory properly.

I admit, I may have the wrong idea here, but the fact that quitting and restarting Lightwave (after rendering a scene with image sequences) allows me to render a frame from that same scene with over a GB less memory used, would seem like a memory leak issue. I don't expect a package as complex and long-term as Lightwave to have no memory leaks at all, but this seems like quite an obvious one to pin down.

Surrealist.
07-26-2008, 05:14 AM
OK. Dumb question, but I have to ask. Are you using the image viewer or the render display when rendering?

DiscoBurgess
07-27-2008, 05:42 AM
OK. Dumb question, but I have to ask. Are you using the image viewer or the render display when rendering?

Just the render preview. Why? I guess the image display would eat memory, but that still wouldn't explain why it only happens with image sequences.

Surrealist.
07-27-2008, 05:56 AM
I have no idea. You could try and see what happens.

DiscoBurgess
07-27-2008, 07:48 AM
I have no idea. You could try and see what happens.

Try what, turning off render preview? Trying that now, getting the same result. Good idea, though.

Surrealist.
07-27-2008, 11:13 AM
Yeah. Had to ask. You never know. Just worth a shot :)

Exception
07-27-2008, 12:10 PM
send in a bug report, if you will.
It shouldn't use more memory after 100 frames than after 1. Except if you're using GI caching.

DiscoBurgess
07-27-2008, 02:17 PM
send in a bug report, if you will.
It shouldn't use more memory after 100 frames than after 1. Except if you're using GI caching.

Happy to. I'm just... surprised that this really is a bug. I assumed there'd be plenty of other posts with people having run into this themselves, but... apparently not. Either it's just my machine, or I am, somehow, the first person to discover it.

Exception
07-28-2008, 06:44 AM
Happy to. I'm just... surprised that this really is a bug. I assumed there'd be plenty of other posts with people having run into this themselves, but... apparently not. Either it's just my machine, or I am, somehow, the first person to discover it.

Have you tried it 9.5 beta? That would be useful to know.

DiscoBurgess
07-28-2008, 06:55 AM
Have you tried it 9.5 beta? That would be useful to know.

Urf, no. Not gotten a chance to hop onto this beta cycle at all. Guess I should install it. :( Or any other kind soul already running it could go ahead and make a scene with some image maps on sprite hypervoxels, and tell me if they're seeing memory usage increase with every frame. The more HVs you have, the more noticeable the effect.

Worst part is, I'm fairly sure I've discovered this way too late to get it fixed in 9.5 (if it's still in there), so to have this fixed (and I could do with it... tried a more ambitious scene yesterday... the switch to 8bit only goes so far) I'm betting I'm gonna be paying out for the next version of Lightwave. Grrr. Oh well, that's the way it goes in software development, unfortunately. Who knows, maybe there'll be a 9.5.1.

Exception
07-28-2008, 07:03 AM
I can't test it as I'm rendering something important, but you should send it in as a bug report anyway, and really make it a decent report, with examples, content to reproduce and everything. If you don't, it won't be fixed... if you do, there's a fair chance someone will look at it.

Also post your report here, so I can look at it when my render is done.

DiscoBurgess
07-28-2008, 07:42 AM
I already made an example scene, just need to write some copy to go with it. Will hopefully get time in the next few days. Hopefully today, actually.

bobakabob
10-01-2008, 02:57 PM
I'm having similar problems, unprecedented since using Lightwave 5.5. Can anyone confirm there are / were memory issues in Lightwave 9.3 with image sequences? I got as far as 150 images out of 500 before LW choked.

Guess I should upgrade to 9.5 but right now can't be doing with the hassle.