PDA

View Full Version : virtual memory and LW 8



JML
04-18-2004, 07:26 PM
does LW 8 use virtual memory on pc ?

in LW8 ,
let's say I have for example 200 spot lights with each 10 000 as shadow map size and have only 1 of them checked in the scene editor.
If I try to render this scene at huge resolution,
is it going to tell me " not enough memory to create frame buffer" ?
or is it finally going to use virtual ram ?

a reminder, right now with LW7.5c

if you have all those spot lights checked off except one of them, when you will render
LW will create room in the frame buffer for all those 200 spot lights! even if there are 199 of them checked off in the scene editor .
not even 50Gb of ram would help to render that!

is this going to be fix ?

Beamtracer
04-18-2004, 09:14 PM
Lightwave has always been able to use virtual memory in Mac OS X. Do Windows applications need to be programmed to take advantage of VM?

JML
04-18-2004, 09:44 PM
did I talk about macs?
does the osX version of lightwave auto-programmed itself with VR?

i'm talkign about LW and memory management when it renders.

do a test on your computer,

create 200 spot lights
each of those spot lights have 8 000 in shadow map size
turn all of them OFF in the scene editor except 1

try to render at 6000*6000

let me know if it renders
it won't (not enough memory for shadow buffer)

now delete 199 lights (which were turn off)
and do the test again. (so with 1 light alone)
it will render

the first test means that LW will create space in the ram for 200 shadow maps instead of 1
(200 * 8000shadowmapsize = alot)

this example is huge to be obvious.

but this problem is annoying when you have a few millions polygons and you have a huge spot light for different views
in your scene
5 views , 5 huge spot lights, when you render 1 view with 1 selected spot light, it will create room in the frame buffer for 5 spot lights. wrong !
if you have even more views with custom lights, or when you render that high Rez, you can't.

but this problem happens to people who make huge resolution
render with big scene.
i'm sure people who renders for tv doesn't have this problem much.
(smaller rez)

wacom
04-18-2004, 10:56 PM
Originally posted by jmnewtek
does LW 8 use virtual memory on pc ?

in LW8 ,
let's say I have for example 200 spot lights with each 10 000 as shadow map size and have only 1 of them checked in the scene editor.
If I try to render this scene at huge resolution,
is it going to tell me " not enough memory to create frame buffer" ?
or is it finally going to use virtual ram ?

a reminder, right now with LW7.5c

if you have all those spot lights checked off except one of them, when you will render
LW will create room in the frame buffer for all those 200 spot lights! even if there are 199 of them checked off in the scene editor .
not even 50Gb of ram would help to render that!

is this going to be fix ?

OK...I understand that the memory issues need to be fixed...but why in the hell would you use that many shadow mapped spots with that hi of rez shadow map?

Architook
04-19-2004, 12:01 AM
LW certainly does use virtual RAM.. it's an OS thing, not an application thing. But the real question being asked I think is will LW 8 handle self-paging for large large scenes? Both Windows and OSX have a 2GB (in some case 3GB) memory limit per process. This is regardless of how much page file you want to give the OS.. it's the absolute Max and adding more RAM chips or hard drives won't help.

You can really hit this 2GB limit with very large scenes. Don't laugh, this isn't stupid, I can fill my architectual scenes with enouigh highres image maps and cloned objects to hit 2GB in a flash, and it's not easy to optimize the scene smaller.

There are two answers to this problem, LW 8 can do self paging, stuff like "Oh, I need this image, but just this corner of it, so I'll keep only that corner loaded in RAM, and leave the rest on disk." This is tough but NECESSARY for supporting large scenes. Pixar does this ALL the time for texture maps.

The other (easier in some ways) answer is to make LW a 64 bit application, and run it on 64 bit Windows on an Athlon 64. That OS does NOT have a 2GB limit [it's terabytes!] so there's no problem.
OSX I'm not sure about but I don't think OSX is 64 bit yet.

Is either solution in LW 8? I don't know.

ingo
04-19-2004, 12:58 AM
Originally posted by Architook ...You can really hit this 2GB limit with very large scenes. Don't laugh, this isn't stupid, I can fill my architectual scenes with enouigh highres image maps and cloned objects to hit 2GB in a flash, and it's not easy to optimize the scene smaller. ....


That happens to me often too, a simple solution will be that instead of a dumb error message (which indicates a lazy programer) LW has to automatically switch from rendering in one pass to render in stripes. So in the beginning of the rendering it calculates what fits into memory and renders one stripe and save the finished stripe as temp files to the HD, going on with the next...; for example like Electric Image does it. That way you can render very high resolutions without problems.

Just my 2 cents

Beamtracer
04-19-2004, 01:36 AM
Originally posted by jmnewtek
did I talk about macs?
does the osX version of lightwave auto-programmed itself with VR?My friend, 64-bit computing is the only way to go to fix this problem!

JML
04-19-2004, 08:48 AM
well , more ram would be the best but my company is not going
to spend thousands of thousands of dollard to put 8 Gb on each render nodes !
If LW would manage memory better like Architook and ingo said,
you would not need that much ram to do high rez image

saving temporary files on the harddrive is the way to go!

ingo and architook seems to be doing the same kind of thing we are doing here. huge architectural renders.
so they understand the problem

beamtracer, what kind of field do you work in ?

tektonik
04-19-2004, 10:12 AM
pffff same resolution vs memory here...

we do architectural renders exclusively... and we feel quite limited by this BUG of store whatever info in the frame buffer...

we grew dependant on fprime in a few weeks but man what a could shower... the RAM problem is worse because each program (LW and fprime) fight for the same real physical ram pie...

it sucks big time

cause i often need 8000 pixels renders for big on-site panels and any bozo with max or even Electric image can chug along at these resolutions

PS I have a dual xeon with 1.5 gig rambus and it is enough for other programs so don't tell me to buy more :)

JML
04-19-2004, 12:13 PM
glad to see I'm not the only one with this problem,

I hope they will fix that soon, it's critical .

toonafish
04-19-2004, 03:43 PM
try spider ( check Flay for that ). I had to render a 14 K image a few weeks ago and spider rendered it with no problems it took a week to render on several rendernodes though. It splits the image into as many pieces as you like and afterwards it stitches them back together again. Well, if you have enough RAM for that ;-) It did not with my 14 K image, so I had to do it in photoshop after all.

Fish
http://www.toonafish.nl

DeltaF86
04-19-2004, 05:15 PM
Why don't you do compositing if you need to do that much rendering. You don't have to do it all at once. I'm not saying this is a solution, but it will help you really need to render out such a big scene.

Meaty
04-19-2004, 05:48 PM
Originally posted by wacom
OK...I understand that the memory issues need to be fixed...but why in the hell would you use that many shadow mapped spots with that hi of rez shadow map?

yeah, i had the same question... I have no idea how LW handles the images in the shadow maps, but realize that one 8000x8000 image in tga storage in a mere 32-bits requires 244 MB of storage. i believe tga is an uncompressed format, so it would probably take up something similar in RAM... multiply that by 200 and... well, there is your answer. Even in grayscale it is 61MB

(61 * 200)/1024 = 11.91 GB Like I said, I am sure LW probably does something to save on memory here, but geez, you are way out of the ballpark in terms of expectation I think.

Beamtracer
04-19-2004, 06:36 PM
The problem with breaking the image into smaller parts, or saving temporary files to the hard drive (which would have to be continually loaded and saved) is that it would slow things down. It may take away the advantages of having a render farm.

64-bit hardware can be advantageous, even when using 32-bit software.

Mac OS X (Panther) has a 64-bit kernel. The OS can access 8 gigs of RAM. Even though the software is 32-bit, each 32-bit app can access its own segment of RAM.

In the case of 32-bit Lightwave, it gets 2 gigs of RAM to itself. Any other 32-bit software also gets at least 2 gigs of RAM, separate to the OS.

I haven't tried the beta version of Win64, but I assume the better RAM allocation would also apply here, when combined with an AMD64 processor.

Architook
04-19-2004, 06:43 PM
So OSX doesn't help, since applications still can't access more than 2GB.

To be honest, there's still problems even at LOW resolutions, if your basic scene and object data hit that max limit.. and I get that mostly with cloned objects and lots of (giant) image maps. So the problem first shows up as not being able to render print resolution, but can easily extend down to can't render ANY resolution.

This is not a LW specific problem, XSI has the same issue with its native renderer. Can't say for Mental Ray. Renderman does NOT have this issue though, they do self-paging pretty effectively.

Anyway, Can't wait for an Opteron version of LightWave! I don't care if it's Linux or WIndows 64. AN OSX version would be cool too but Newtek would have to wait for Apple to add 64 bit application support first.

JML
04-19-2004, 07:14 PM
Meaty

read all the posting , not just the first one and you will understand.
it was a huge test to see if spot lights turned off in the scene editor would take space in the frame buffer.

as you said 8000*8000 shadow map size take a lot of memory,
so it's important that LW doesn't include spot lights which are off in its frame buffer before rendering.


anybody tried renderman with LW ?

Meaty
04-19-2004, 08:35 PM
I see what you are saying... that is weird... probably a simple fix too...

But as a workaround for rendering lights off camera, you could use the spreadsheet to make the 199 lights have a shadow map size of 16 (smallest possible), in addition to unchecking it in the scene editor. This will dramatically reduce the overhead. Would this help in your particular situation?

JML
04-20-2004, 11:01 AM
that's right meaty, this trick would work.
but that would be cool not to have to do that every time.. (if it would be fix)

cool work meaty, I saw your web site,
is it your own company ?
I work in silverspring,MD

Meaty
04-20-2004, 11:41 AM
No, not my own company. It is actually an engineering firm. Most of what we do are simulations and visualizations to support our engineers/analyst presentations.

We are a three man strong multimedia department. Jack of all trades, master of none =]