PDA

View Full Version : Is it possible to render a seperate FiberFX or Hair pass in LW?



Curly_01
04-12-2014, 01:19 AM
Hi,

Since you can not use FiberFX with Octane Render, is it possible to render a seperate FiberFX or Hair pass in LW? Or can you do it with the Janus renderpass plugin?

Greenlaw
04-12-2014, 11:53 AM
Yes, you can. Rendering FiberFX hair in a separate pass was what I did for in the DmC (Devil May Cry) theatrical trailer we created in the Box a while back. It's a little tricky though as FiberFX doesn't seem to be designed with proper compositing in mind, and you have to sort of hack around the features if you want fine fibers with good anti-aliasing, and if you want the cast shadows in yet another pass. (This isn't limited to FiberFX--I had to pull similar tricks with Sasquatch too.)

I'm a little busy at the moment but I'll return later today with explanation.

G.

Greenlaw
04-12-2014, 12:00 PM
That said, without going into detail, the one thing that might be an issue is that the AA between Octane and the LightWave renderer might be different, and you may find a gap or a matte line along inside edges of the character. We sometimes saw this problem with Sasquatch.

(I'll still come back with more info later.) :)

G.

hdace
04-12-2014, 06:46 PM
I'm also doing this, not by choice. I have a male character with long hair and a long beard and FiberFX causes the scene (which is very big and complex) to hang for several minutes before rendering each frame. This was obviously unacceptable so we're doing it in a separate pass. I found that I could actually improve the look of the hair and beard by lighting them slightly differently, but I'd still rather do it together if possible.

Greenlaw will probably come up with lots of ideas I haven't thought of, but the main thing you'll find is that there's an output tab in the FiberFX window. If you turn this on, and turn OFF output in the render globals output tab (allowing you to pare down the scene to a very simple version with skin nodes turned off, etc), then you'll get a perfect (at least I've found it to be perfect) separate hair element for compositing later.

The biggest problem is the shadow the hair should be casting on the body, clothes, etc. In our scene the hair and beard guides in the master scene are animated using mdd's loaded into ClothFX that were generated using Bullet. The guides can cast the shadow without being rendered. I find this works quite well. Only problem will be reflections, which I haven't encountered on this project yet, but I know it'll come along sooner or later. The way you do this is by rendering the master scene (with FiberFX removed entirely) with Render Lines turned on in render globals render tab. Select your guides, then open object properties render tab, unclick self shadow and receive shadow but leave cast shadow on. Then click unseen by camera. And then one final trick, you have to go to the edges tab, delete "1.0 pixels" in the particle/line thickness field, and type "-0.01 meters". I've never quite understood why this works, but it is in the documentation somewhere. It gives the lines enough thickness to generate fairly appropriate shadows.

Greenlaw
04-12-2014, 08:00 PM
Hi, I'm back. :)

I have to keep this quick but if you have questions, feel free to ask:

First, here's the way it's supposed to work:

Under the FiberFX Output tab, enable Save RGBA. This will save the FiberFX fibers in its own layer without the character, and save the regular LightWave render minus FiberFX (but with the cast shadows.) The 'gotcha' here is that your FiberFX layer will not have anti-aliasing. I think I was told that this isn't a bug but if that's the case, I'm not sure what this render is good for. This is too bad because if it actually worked, it would be so easy and productive to output FiberFX in its own layer.

Anyway, all is not lost. Here's my 'hackaround' that I used back when we made DmC:

1. First let's output FiberFX in its own layer WITH anti-aliasing. To do this, select your geometry and in the Object Properties under the Render tab, enable Matte Object. Set the color to 100% black and set Alpha Channel to Black. Also, make sure the background is black.

2. Render the scene. The result should be a FiberFX only pass with normal LightWave AA.

3. Optionally, set the Alpha Format to Unmultiply Alpha and set the appropriate setting for this in your compositing program. If you don't do this you may find your fibers looking much thicker than in a straight single-pass LightWave render. This is what I had to do for DmC--if I have time, I'll post examples of why this option is preferable.

Now, about those cast shadows, here's another 'hackaround':

What I did for DmC was to create shadow catcher versions of the geometry and output a shadow mask to use in composting. My preferred method is to set the Alpha Channel property in the Surface Editor (under Advanced) to Shadow Density--this will replace the normal Alpha Channel with a mask for cast shadows. Other user have told me they like to use the Shadow Catcher node, but personally I find Shadow Density masks are a bit easier to set up and composite with. The choice is yours.

To keep the fibers out of the shadow layer, you'll need to 'hide them' from the camera. you can't use the normal Hide from Camera option though. But what you can do is this: use Save RGBA in FiberFX to trick Lightwave. (Yes, this feature is actually good for something...even if this is not the intended use!) :)

Anyway, enable Save RGBA in the FiberFX panel and set it to save a dummy file--I usually just call this file 'dummy' and save it as jpeg to save disk space. After I'm done rendering the shadow pass, I delete these dummy files--it's just junk so don't worry about it. What's useful about using FiberFX Save RGBA is that is prevents FiberFX from rendering fibers but it still allows FiberFX shadows to render in the normal LightWave render, so in this case the result will be a mask of the shadow in the Alpha Channel of your LightWave. You can can disregard whatever data is in the RGB channel but use the Alpha channel as a mask in your compositing program to darken or color the character's skin between the skin and fiber layers.

The big 'gotcha' with this is method that the shadows rarely look quite right without adjustments made in the LightWave scene. Often times, I found I got better shadow pass results if I minimized the lighting to a single Key light, and sometimes I had to adjust the angle of the light to favor the camera. That might sound like a bug but it's apparently a common problem with many hair/fur system--sorry, I don't know why this such a problem but I guess it must be very difficult to program. FWIW, I ran into this same issue years ago whenever I had to output shadow passes with Sasquatch, and when I was researching how it was done in other programs, I learned that Shave and A Haircut in XSI also had this problem. BTW, you can use VPR to check the shadow quality and angle--checking shadows in VPR is usually interactive since you're only viewing the cast shadow and not the fibers.

Nowadays, I don't bother rendering the shadows in a separate pass--I just render the 'bald' character with the fiber shadows baked in. This may limit how you can edit the shadows in comp but it also looks more correct to begin with so you may not need to tweak it.

For our current 'Brudders' short, I'm just rendering the fibers with the character pass. If I need a fiber mask, I render a pass with the matte objects, and I turn off all lighting/raytracing effects so I can render this pass very quickly. This limits how you can work with the fibers in comp but for our little LGD productions, this workflow has proven to be the most productive approach. (Mileage may vary by the scale of your production of course.) This method will not help you when using Octane--I'm just mentioning it here for completeness.

Hope this helps. The one thing I can think of that may mess things up is the difference in AA as mentioned above. But I don't know that for certain, so please let us know how it works out for you.

BTW, here's a final 'gotcha' to watch out for: Don't bother rendering FiberFX animations in 11.6.2. There is a bug that will make the specularity appear to be stuck to the fibers like a texture map. I've already reported this bug and was told that it was being fixed. In the meantime, you should be okay rendering FiberFX in an earlier version of LightWave though I'm not sure when this feature got broken so you may need to experiment with different versions. I do know FiberFX rendered just fine in 11.5.1, but I'm not sure about 11.6 or 11.6.1. (We rendered the 'Brudders 2 Excerpt' (https://vimeo.com/68543424) last year in 11.5.1.)

Did I say I was going to keep this short? Sorry, I lied. :p

G.

Greenlaw
04-12-2014, 08:18 PM
I think I'm going to report the 'no anti-aliasing in Save RGBA' issue again. Even though I was told this was normal, I really don't find the result very useful without the same anti-aliasing as rendered through the selected LightWave camera. Maybe they can add 'Apply AA' as an option?

I hope other LightWave users will submit this request too.

G.

Curly_01
04-13-2014, 01:52 AM
Can't you just render out a scanline renderpass with a toonshader on the hair. In the past I've experimented with 3Delight and it's very fast for rendering hair? Can't we just render a toonshader on the hair, invert the pass and add blur for the specular highlight on the hair?

Curly_01
04-13-2014, 01:59 AM
Or just do a simple QuickShade renderpass and mask the rest out?

Greenlaw
04-13-2014, 02:35 AM
I've never tried toon shading FiberFX--is that even possible? FWIW, I've toon shaded Hypervoxels in a past project, so maybe you could do that.

Let us know if you figure out how.

G.

Curly_01
04-13-2014, 03:00 AM
http://www.youtube.com/watch?v=fW_0oRoWJyA

Greenlaw
04-13-2014, 03:17 AM
That's kinda neat. Although when you said 'toon shade', I was thinking more traditional cel-shaded outline/fill--but I guess for that look it would be better to go with modeled geometry rather than fibers. Not a critique, just a different graphic style.

G.

Oedo 808
04-13-2014, 05:07 AM
Thanks for the furry tips, Greenlaw. One of these days I'm going to render out something nice and hairy!

Dodgy
04-13-2014, 08:16 AM
Hey greenlaw, are you saving in 24 or 32 bit from the ffx panel? Only I imagine in 24 bit the hair looks aliased as its supposed to be pre alpha which looks right in 32 bit. Just doing a test, it looks antialiased to me in 32 bit.

Greenlaw
04-13-2014, 12:29 PM
Hey greenlaw, are you saving in 24 or 32 bit from the ffx panel? Only I imagine in 24 bit the hair looks aliased as its supposed to be pre alpha which looks right in 32 bit. Just doing a test, it looks antialiased to me in 32 bit.
Hi Dodgy,

Thanks for checking this for me.

That's really surprising that you're getting AA with Save RGBA though. I did a quick test last night before posting to be sure, and I can definitely see a difference between the two renders (normal LW render vs Save RGBA,) and I got the same results I saw a year an a half ago when setting up FiberFX for DmC. Here's what I'm seeing now:

121429

The image on the left is fibers only pass rendered through the Perspective Camera using a sphere with Matte Object Black and Alpha Channel Black enabled. The image on the right is the Save RGBA pass. The image on the left matches the regular single pass render whereas the image on the right is chunky and definitey different. Maybe there's some AA there but it's not very much. (IMO, I really think there isn't any AA and what looks like 'some' AA is just the differences in the shading.)

These should be 32-bit EXR renders. At least, that's what I typically render by default. Let me check my settings again to be sure I didn't render a lower depth by mistake. I'll post a followup when I'm sure.

Anyway, the dev did tell me a while back that Save RGBA either does not or cannot use the 'in-camera' AA which is the why there is no AA (or maybe I misunderstood and it's just different AA when you use Save RGBA?( Sorry, it's been a long time since we had that exchange so I'm definitely paraphrasing the developer.

In any case, the Save RGBA render looks awful compared to the 'in-camera' render and isn't very usable for compositing--the fibers are very chunky and may as well not have any AA--and I mean it's like that over the whole render, not just along the outer edges.

Save RGBA also has issues with properly sorting fur/hair passes that was supposed to be in front and behind an object--like hair on the head that hangs in front of the face and behind in a 3/4 view--again, I was told this is because it's not doing the render the same way it does when through the camera.

G.

Greenlaw
04-13-2014, 12:43 PM
I should re-run this test using a super closeup of just a few fibers. Obviously, there's a difference in the renders but this new test should tell me if there's any AA being applied at all when using Save RGBA.

(I guess I'm officially back to FiberFX testing again.) :p

G

Greenlaw
04-13-2014, 01:31 PM
I just did a super duper closeup of a single fiber using both methods, and here's the result:

121430

It does appear that there is some level of AA being applied when using Save RGBA (right image) but the quality is far inferior to what I'm seeing in the 'in-camera' render (left.) This would explain the 'chunkiness' I've been referring to when using FiberFX Save RGBA.

I think I need to just resubmit this fogbugz as this would be an awesome feature if it worked correctly. (Or if I'm just using it incorrectly, they need to update the manual to explain how to use this output properly.)

G.

Greenlaw
04-13-2014, 01:33 PM
Actually, the render on the right might be even chunkier than shown here because some of the apparent 'smoothness' seen in the image is coming from Photoshop's scaling interpolation.

hdace
04-13-2014, 08:07 PM
Hi, I'm back. :)

I have to keep this quick but if you have questions, feel free to ask:

First, here's the way it's supposed to work:

Under the FiberFX Output tab, enable Save RGBA. This will save the FiberFX fibers in its own layer without the character, and save the regular LightWave render minus FiberFX (but with the cast shadows.) The 'gotcha' here is that your FiberFX layer will not have anti-aliasing...
G.

Greenlaw, it's always a joy to read your long highly scientific posts. I know I'm going to sound like an ignorant heretic, but I have to say, I've never been too bothered about hair AA. At least out hippy character simply doesn't appear to need it. I've compared the full scene renders with the hair rendered in the master scene (despite the hang problem), and the results of comp when they're rendered as separate passes using FFX output, and I can see almost no difference (despite the hair and the rest of the scene having been AA'd with adaptive sampling threshold set to 0.01 and 12 sample passes!) between the two. Mind you, they're all medium shots. No big cu's. I think I'm not as sensitive as you are to such subtleties, which surprises me because I thought I was kinda too anal about such things. I did actually use your in camera solution (which I worked out myself) a long time ago before the FFX output was implemented. It's a bloody pain in the neck, but you're obviously right that it'll keep the AA working appropriately.

But the big problem we have is that FiberFX is causing a render hang. It doesn't happen when we use a pared down version of the scene (the character only). Something is causing a conflict and we haven't been able to identify it. So using the dummy file method wouldn't work for us since FiberFX would be on and it absolutely has to be off for us to get any rendering done. You said, "I just render the 'bald' character with the fiber shadows baked in." I actually don't understand what this means. Would you be so kind as to explain it to me? As I said earlier, I'm using the guides to generate the shadow onto the bald character. Is that a really crap method? I've liked the look of it so far, but if your baking method is better I'd love to use it.

Dodgy
04-13-2014, 09:10 PM
121435121436What kind of AA are you using? The settings I mean? Because that "AA"'d image on the left looks more like it's been blurred, probably by the gaussian reconstruction filter, which is the reason I don't like that filter much. Even if you set the AA to 1 with no adaptive I find it blurs the render slightly. If I render with classic/classic with reasonable settings I get these two renders which are pretty much identical in terms of AA. If there was no AA on the FFXout file, I'd expect it to look a lot worse than it does. In your close up images, the right one is the one I think has AA applied more correctly, I'd only really expect AA to affect the pixel next to a change in pixel colour, not 2 or 3 or 4 pixels away, that's more of a blur. 121438 This circle drawn in photoshop shows what I mean. The edge is only antialiased 1 pixel either side of it, not multiple pixels, as an edge can only cut 2 pixels, not 3 or 4. I agree it's not matching the render AA for your settings, but it is certainly doing some antialiasing otherwise it'd look like a picture from the 80s. Blur the output from the FFX filter if you want the same result. If you try a gaussian AA on just a static picture as the background you'll see it comes out blurrier than when it went in, so maybe try loading the output from FFX through LW as a BG image sequence with the gaussian AA filter on, and you'll get the same result as the one from the straight LW render.

Greenlaw
04-14-2014, 07:04 AM
Thanks again Dodgy...that could be what's going on here with my renders. I normally use Classic for all-CG work, but I'll use Gaussian to get a little softness when I'm comping to a film plate. I might have had Gaussian enabled out of habit since I've been doing a lot film fx work lately--I'll check the settings in that scene as soon as I get the chance. That said, I wish FiberFX Save RGBA used the same AA as the camera regardless.

But if sticking with Classic mode will get get the AA to match exactly, that's all that really matters to me. I'll let you know if this does it for me.

G.

P.S., Just remembered: we always used Classic mode in the Box and we definitely got bad AA with Save RGBA, at least compared to the in-camera FiberFX renders--so now I don't know again. But that was quite a while back and many FiberFX versions ago so I need to revisit this issue.

Also, there was the matter of incorrect fiber sorting when we used Save RGBA, which had nothing to do with AA. But again, that was some time ago--need to revisit and retest. :)

Greenlaw
04-14-2014, 07:07 AM
Oh, to clarify the additional blurring you're seeing in the above image, as mentioned in the followup post, some of that is coming from Photoshop scaling. I should have just done a screen cap of a zoom-in instead of scaling the image. Sorry for the confusion.

G.