PDA

View Full Version : HD Instance or Separate Renders?



bluesmike
09-08-2009, 06:20 AM
I haven't done many really large scenes in my LW life, but this project is huge for me. It calls for quite a few full 3D trees, moving people, animated fountain (textures not particles), moving vehicles, etc. And I am experiencing Out of Memory While Rendering messages.

I'm running LW 9.6 on a Pentium Quad Core 2.66 GHz, 4 GB of ram on XP. The scene currently has just under 4 million polys and around 3 million points, 107 images, and the scene is not quite finished. I've optimized the scene and the radiosity settings as best I can (yes, I've read and re-read the excellent guide to 9.6 radiosity by Except).

I've done a few experiments where I've isolated just the trees -- renders fine. Isolated everything BUT the trees -- renders fine. Combine them and WHAM! memory out.

I see that I have at least two obvious solutions and I was wondering if there is anything to favor one over the other. (1) I could purchase HD Instance and hope that that solves the memory hit I'm taking on all the trees (I have three different trees and the rest are clones).

(2) Or, I could try to render out a pass with just trees, and a second pass with everything BUT the trees and composite them later. What I'm concerned about is the extra labor hit I'd take on just setting this up -- this scene calls for a lot of interaction with the tree shadows and fear this would be tricky to set up.

Any suggestions from those of you who've worked with large scenes in the past?

Mike

Lightwolf
09-08-2009, 06:23 AM
Well, there's also:
3) Get more memory (assuming you're running a 64-bit OS and 64-bit LW) as it's dirt cheap nowadays.

Otherwise I'd go for 1), especially once the passes start overlapping you'd still need the "other" geometry for the matte objects.

Cheers,
Mike

bluesmike
09-08-2009, 06:33 AM
No, I'm not running 64.

Mike

Lightwolf
09-08-2009, 06:36 AM
No, I'm not running 64.

Mike
Well, in that case banging in another 4 or even 8GB can't hurt - if y<our board can take it that is.

Cheers,
Mike

JBT27
09-08-2009, 07:21 AM
Also, if you are considering the HD Instance approach, although it is stunning and vital for many things, with foliage, especially dense foliage, flickering is an issue. We're having to deal with this at the moment - AA settings are going through the roof with little impact on quality. This is, in fairness, a LW renderer issue (indeed an any renderer issue :)) as instanced detail in the background reduces to less than one pixel.

We're back-tracking to more simplistic approaches for dense foliage, before we get too old waiting for the grinding slowness of LWSN, and the aliasing flicker of too much detail.

Foreground instanced detail is suffering as well - don't know why.

So just a 'warning' to perhaps also look to simpler approaches. The multi-layer/multi-pass approach is a good one, and as ever, gives alot of control, especially if you're wielding exrTrader and rendering to the OpenEXR format.

More RAM wouldn't hurt either, and 64 bit ..... :)

We switched recently - don't use it for everything, but essential for some shots.

Julian.

OFF
09-08-2009, 07:37 AM
and another good solution - infiniMap:
http://www.db-w.com/

biliousfrog
09-10-2009, 04:54 AM
The flickering issue can often be solved by rendering at double size and reducing in post (lower the AA to compensate for render times). By increasing the amount of pixels you decrease the possibility of flickering on/off when a polygon reaches 1 pixel because it would then be 2 pixels.

geo_n
09-10-2009, 05:34 AM
you could render out in passes so you won't run out of memory.
I did a test in lw to see how much resources it could handle(compared to max) and didn't get any issue even with multiple animated characters with bones. But to get more people I would use mdd and hdinstance.
http://www.vimeo.com/6476478
The good thing with that test is I can swap out any of the element if there were changes without rerendering the background. So with rendering trees,etc I would group them to different render passes.

JonW
09-10-2009, 07:09 AM
When you are ready to render, using Bounding Box in layout view ports can free up a stack of ram. You need to save the scene with the view ports using bounding boxes, close & re-launch (you can if you are very quick with the mouse, when the scene loads change each port to BB & not too much ram is consumed).

I had a scene & with just one view port with textures LW was using another 2gb of ram.

Or get XP64 or similar, & as much ram as you can afford, its very cheap now, & you definitely don’t want the computer using virtual ram when rendering.

Finally but not least, (& I would not try to set up during a job)

Screamernet on one computer works well & you don’t even need to load the scene & you save a stack of ram. Again a recent job, 7gb F9/F10 render& about 4gb when rendering with Screamernet.

bluesmike
09-10-2009, 09:09 AM
Thanks a lot for all your suggestions. I am testing some of them out now before I commit to spending money on HD Instance. I had a friend also suggest I try using ScreamerNet on one computer to render. I've never used it before, so that's a new experience, but I'll give it a try.

Again, thanks. I'll repost here if I have some success. Worse case, I might have to use a commercial renderfarm, but have no idea what sort of cost I'd be looking at, and I don't know if my budget can take it.

Mike

biliousfrog
09-10-2009, 10:49 AM
Personally, I couldn't bare the thought of setting up screamernet so I bought ButterflyNet Render instead, it sets it all up for you in no time and the personal edition is cheap too.

http://www.liquiddreamsolutions.com/web5/index.php?option=com_jumi&fileid=4&Itemid=159

JonW
09-10-2009, 07:18 PM
If you use a render farm, you will need to be fanatical & get the scene exactly right or its going to be a very expensive exercise. Say, render every 10th frame etc to iron out all the problems, plus maybe the whole thing at a reduced size.


Check out http://3dspeedmachine.com/?page=3&scene=39 it gives a fairly good indication of relative speeds between computers.

If you are running only the Pentium, get a new absolute Bare Bones Box, MB, i7 920, 6 or 12 gb ram, basic HD, are the key items, you can even put in a dog of a graphics card for now as LW doesn’t use them for rendering anyway & upgrade it later if you decide to use it as a work station.

As long as you have enough ram the CPU is really the only issue, & a 920 could make the world of difference, it will give you the best Ghz per dollar per box. Especially with radiosity the i7 & x55xx CPUs are a leap ahead even from the last CPUs.

A BBB can be really cheap & nasty, its only a render node & no one is going to see it.

PS. Radiosity on trees you can quite often get away with brutal figures RPE 24 SBR 8 AT 90 MinPS 4 & no will ever know & it can save a lot of time. Even turn off radiosity on 2 out of 3 trees if they are in a group. Turn off Affect Specular for all lights in most cases you wouldn’t notice the difference.

JonW
09-10-2009, 07:49 PM
Billousfrog’s suggestion of doubling the size is an seriously good idea & will solve a multitude of problems.

Ok you have 4 times the area but you cull the AA, & at the same time, as he pointed out, flickering at the 1 pixel size is solved. It doesn’t matter so much with flicker at this double size, & its usually less anyway, because when you halve the size the flicker is usually nailed, or its so small its not an issue.

You can usually render a frame quicker at 4 times the size with lower settings, & frame to frame you have better quality so there is a secondary saving. With the final size then halved this is creating the AA & you can add a bit of blur in post if required, its now looking better & rendering quicker than if rendered at this size in the first place with very high AA & most likely higher radiosity settings.

Turning up Mipmap Strength to say 5 if there is flickering on textures is an option.

geo_n
09-10-2009, 08:54 PM
Personally, I couldn't bare the thought of setting up screamernet so I bought ButterflyNet Render instead, it sets it all up for you in no time and the personal edition is cheap too.

http://www.liquiddreamsolutions.com/web5/index.php?option=com_jumi&fileid=4&Itemid=159

I've tried the demo of bnr. It fails to render for me and it has to do with textures. Can it work like sending all the files to the nodes temporarily and rendering it there or does it have to access a sort of network drive that can be seen by all nodes?
I'm just used to backburner I wish lw had it.

adk
09-10-2009, 09:13 PM
I've tried the demo of bnr. It fails to render for me and it has to do with textures. Can it work like sending all the files to the nodes temporarily and rendering it there or does it have to access a sort of network drive that can be seen by all nodes?
I'm just used to backburner I wish lw had it.

It needs a shared path (as described in the vids/docs) that all nodes can see and access. Once you set it up properly it works like a treat.

On my first attempt I also failed (had black frames) but when I re-saved the scene (and anything to do with paths - image saving etc) with absolute paths it worked just fine.

hope that helps.

biliousfrog
09-11-2009, 04:08 AM
Yeah, I have all my current projects on a shared RAID array (P) and the screamernet files are stored on another shared directory (R). Every node has those same locations mapped to the same network drive letters.

bluesmike
09-11-2009, 10:22 AM
JonW, thanks -- I'm already getting some promising results from just doing the bounding box "trick" you described. At least now it's actually rendering -- I did a test and got all 200 frames of the test, where earlier it would "memory out" at only a dozen or so frames. That's some progress! I will try a few other things - I like the idea of increasing the size and reducing some of the other settings to reduce flicker and speed up rendering.

Thanks all for the great suggestions. This has been really helpful.

Mike

JBT27
09-13-2009, 01:12 PM
Billousfrog’s suggestion of doubling the size is an seriously good idea & will solve a multitude of problems.

Ok you have 4 times the area but you cull the AA, & at the same time, as he pointed out, flickering at the 1 pixel size is solved. It doesn’t matter so much with flicker at this double size, & its usually less anyway, because when you halve the size the flicker is usually nailed, or its so small its not an issue.

You can usually render a frame quicker at 4 times the size with lower settings, & frame to frame you have better quality so there is a secondary saving. With the final size then halved this is creating the AA & you can add a bit of blur in post if required, its now looking better & rendering quicker than if rendered at this size in the first place with very high AA & most likely higher radiosity settings.

Turning up Mipmap Strength to say 5 if there is flickering on textures is an option.

I must have things seriously off with my scene, because I also tried this, but the flickering was still there. And I'm not talking about minor-fix-it-with-an-AE-plugin flicker - even at four times the area I'm seeing serious artefacting, even on foreground leaves, with the HD Instanced stuff.

Ramped the AA up again, used Overscan, used softer reconstruction filters ..... I can only think I must have set up such a density that no amount of damping of any sort can kill it.

Back to the drawing board - not a bad idea when it gets this bad :)

Julian.

bjornkn
09-13-2009, 03:42 PM
And you are sure that Volumetric AA is on too, Julian?

jdomingo
09-14-2009, 05:36 AM
hi guys, i have a questions about HDi. i have the latest version of it and i am just wondering if there is a way to assign the amount of ray recursion limit and ray precision only to the object being instanced without affecting whatever is in the ray rec limit and ray prec on the render tab? what is happening is, i have to change the setting on these two just to render my scene fast with HDi and if i dont, it just too slow for me.

biliousfrog
09-14-2009, 06:24 AM
Are you using a weight map to control density? There were serious issues with weight maps in the previous '2' versions, I'd suggest reading through the autograss and grass with HDi threads. It sounds like an issue with HDi rather than sub-pixel flickering.

http://www.newtek.com/forums/showthread.php?t=101604

http://www.newtek.com/forums/showthread.php?t=101414&highlight=autograss

JBT27
09-14-2009, 08:24 AM
And you are sure that Volumetric AA is on too, Julian?

Well yes, pretty sure ..... just going through the scene again because it looked good with all that HDI foliage and how I would prefer to do it, even with the long render times.

Definitely using 2.0.7 - just updated last week, but not using any weight maps in this one.

Julian.

JBT27
09-17-2009, 10:00 AM
And you are sure that Volumetric AA is on too, Julian?

..... :foreheads ..... as it turns out - that's exactly what it was! Felt a complete idiot :D

I checked over and over and convinced myself it was on, but I was also bouncing around 14 versions of the scene, and probably switching stuff off and on to figure what had gone wrong.

But clearly I had gone wrong!!!

Thanks for waking me up to that :)

Julian.