PDA

View Full Version : AA on background image gaahhh



clagman
02-04-2010, 08:45 AM
How can I get Lightwave to stop putting AA on background images and such? Or am I screwed?

Captain Obvious
02-04-2010, 09:28 AM
Don't render a background image, or a backdrop at all. Create a 1x1 image filled with a single pitch black pixel, and set that as your background image. Then do a seperate background pass, and comp it.

Rendering your background image in the main pass is simply THE WRONG WAY. Just like rendering directly to a movie file.

4dartist
02-04-2010, 10:30 AM
Definitely agree with CaptainO. If you don't have aftereffects or a composite program, you can always load your frames back into lightwave as the foreground, and put your background back in and re-output. Lightwave can do basic composites like that.

clagman
02-04-2010, 11:10 AM
Well I should have provided more information. Here is the situation:

I'm using Vue and Lightwave on this project. Vue renders Vue content SOOOooo much faster than Lightwave when it comes to AA so the idea is to keep Vue in Vue as much as possible. So here is what I am doing...

I have a small part of the ecosystems being rendered by Lightwave because they are very close to the LW objects in the scene (as in casting shadows on and having shadows cast on them). The rest of the scene (infinite terrrain with ecosystems and clouds etc), is rendered in Vue and comped in AE. The problem I noticed was that the Vue stuff (being a volumetric plugin) was rendering all of its stuff with AA pixels the color of the background. If you use a black background then you get black AA pixels. The alpha contains the black pixels so you are having some real problems getting rid of them.

My current workflow uses skylight as the source for radiosity on my LW objects (volumetric rad is way expensive for what you get from it). So the idea is...turn off Vue scene use skylight and bake radiosity. Then turn on Vue and send to render using the previously rendered Vue frames as a background so you don't get the black or miscolored AA pixels. Works great BUT Lightwave wants to AA the background image. Although it only uses up a few extra minutes I hate the idea of AA something that won't be in the final render (at least for the Lightwave half).

Vue won't respect unpremultiply unfortunately or all this wouldn't be a problem.

In some cases I can pull Lightwave models into Vue for rendering which works fine but in this case my LW objects have nodes for texturing which Vue can't understand so....

Any ideas?

4dartist
02-04-2010, 11:22 AM
Hey clagman.

If you use a black background in Lightwave, and output in render globals is set to 'premultiply', then on your rendered layer in after effects, you must set the interpretation to 'premultiply' and set the color to 'black'. This will get rid of all the dark boarders around your render. (this isn't a work around either, this is how you comp premultiplied stuff that was rendered over a black background)

Another option, that I usually don't use, but there shouldn't be any issues, is to set the output to 'unpremultiply' in lightwaves output tab, in render globals. Then lightwave won't render ANY background colors or composite layers set to the background. Then your edges should be clean too. However I rarely use this option, I find pre-multiplying works just fine when you set your AE layer to premultiply black too.

You can set each asset you load in AE to premultiply or unpremultiplied. When you premultiply you can pick a color, which is usually black, which will adjust for your lightwave black background, it works perfectly really. Sometimes with hypervoxels you may like the look better unpremultiplied, it can make the voxels look thicker.

clagman
02-04-2010, 11:51 AM
Unfortunately, that isn't going to work though because of the Vue/AA/alpha situation. Vue objects include AA pixels inside the alpha so you can't get rid of them by playing with alpha channels in post. You get black dots (or whatever color your background is). Premultiplied or unmatted makes no difference. So you pretty much HAVE to use a matching background plate to fix the AA problem.

Here is a very rough render but should give you the idea. Notice the black fringe on everything. Rendered on a black background with a "solid" created in AE behind it. This is set to unmatted, setting to premultiplied makes it far worse.

The obvious answer is "I wish Vue would just work with unpremultiply output option" but that doesn't work sooooo...

There is another update so maybe this one fixes that problem eh. Fingers crossed. If not it's really not a show stopper anyway. More of an annoyance.

4dartist
02-04-2010, 12:05 PM
Clagman.. don't give up here.

--deleted all this junk--

Forget the unpremultiplied, you don't need it. So put lightwave to Premultiply. Every set of render, except your ultimate background render that will be composited at the bottom, MUST be rendered using an all black background. That doesn't mean your lightprobe, or radiosity image, or any sort of images in the --Backdrop-- tab of lightwave needs to be black. What is does mean is that you need a black image in the 'background image' pulldown in the --compositing-- tab of lightwave.

This way any composite layer, gets black burnt into the edges of your layer uniformly. Using After Effects, this black is simple to unburn from the edges using 'Premultiply w/black'. But it will only work if you rendered against a black background with 'premultiply' set in --output-- tab of render globals.

4dartist
02-04-2010, 12:19 PM
Vue objects include AA pixels inside the alpha so you can't get rid of them by playing with alpha channels in post. You get black dots (or whatever color your background is).

This is normal. That's how 3d programs works. Your color channels and alpha channels both get AA work done on them. Your color channels will have the background's colors 'burnt into the edges'. This is fine, because you basically burn black (background) into the edges, then in comp you set the interpretation to premultiply that black into the edges and it's compensates for it flawlessly.

Lightwave is different because it has an unpremultiply option, which is handy for stills, but really you don't even need it when you are using After Effects to composite.

THIS IS IMPORTANT:
From reading your posts I don't see you confirm this: You MUST use a solid black background in the 3d app in any renders except the render that will be the very back, like your mountains/sky. What Captian Obvious said is something you MUST do when you use premultiply as the option in lightwave. There is always a way to force lightwave to have a black background in the render. Usually the easiest way is to go to the -compositng- tab and put a black image as the background.

clagman
02-04-2010, 12:45 PM
Ok, thanks for the help man. To clarify, when I render standalone Lightwave objects with no Vue and set the output to unpremultiply, I bring the images into AE and set the alpha to unmatted and blammo no halos. I am using black background. Image or standard background color results in the same.

If I then insert Vue objects into the scene this no longer works. Setting to unpremultiply and render import to AE and set to unmatted leaves black halos on everthing. Not much else to it really. Seems a limitation with Vue. So to compensate I have to use background plate and render but want to get rid of AA on it (i'm guessing not possible though).

Alternately I can get the exact result when I render with premultiply and set premultiply (matted) in AE. Black halos on all, but even worse than the image I posted before. All this works as expected so long as you don't render Vue items.

Odd eh. My guess is that Vue doesn't like the idea of rendering without the sky being turned on lol. No big deal. Gotta get this scene to the farm so no time to play with it until next week.

4dartist
02-04-2010, 12:57 PM
Hum..

The summary:
If you set LW/Vue to premultiply, render against a black background using 32bit .png and not separate alpha frames, load 32bit frames into AE, set AE to premultiply that layer in the interpetation panel with a black color, and you still have dark halos, then I have no clue what the problem is. Vue should be smacked in the face with a frozen chicken..

clagman
02-04-2010, 01:01 PM
I'm using EXR_half but ya basically that. So the question is, how can I stop Lightwave from applying AA to the background image haha!

4dartist
02-04-2010, 01:12 PM
Where are you putting the background image in Lightwave? Just in Compositing tab, under background image? I put a huge sky image in there, and set AA passes to 50 and AS threshold to .01 (this is freaking insane AA settings) and it rendered in 1 second. I did see the AA run on the image though, but only for a split second. If you drop a sequence in here is shouldn't be any different.

As to get LW to not AA the background in any shape or form, I really have no clue. Sorry.

clagman
02-04-2010, 01:18 PM
Yep the composite tab. Like you said, it really doesn't add much to the time. So I guess it doesn't bother me THAT much. I'm just being picky about it. Maybe the next LW will have AA settings per object or exclude or something for us retentive types ;^)

Captain Obvious
02-04-2010, 01:20 PM
I don't see the problem. Just unpremultiply as you're comping. Ie, don't use "normal" blend mode, use "over" instead.

I work with layers like this quite literally all the time, and I never have problems with outlines. Just render against a black background and unpremultiply, and you'll be fine.

4dartist
02-04-2010, 01:26 PM
Haha ya. I agree.

I depressed slightly about the vue rendering/compositing. I wonder if other people have had similar issues, and have to resort to rendering against a background, just to force bleed edges with colors similar to what it's going to be composited against. Totally annoying for sure.

clagman
02-04-2010, 01:41 PM
CaptainO. Man, it doesn't work. I wish it were that simple. With the halo you can't comp that way. Changing to premultiply, or unpremultiply against black image or background color black, all the same. The tell-tale problem is that without the Vue integration it works exactly the way I expect it to (using either method). Once you add in Vue stuff black borders that don't go away.

I'm actually starting to wonder if the problem isn't something to do with AE rather than my LW/Vue setup. On this computer sometimes when I work in 32b mode it can act strange.

4dartist
02-04-2010, 02:09 PM
Doing simple tests would help nail it down I'm sure.

If I was going to test it I would do this:

1. Put LW to premultiply and set output to 32bit .png (you can test with exr if this all works ok)
2. Render a pure white polygon (100% lum. and no lights) against black background in LW.
(moving the camera to make a nice soft motion blurred edge and will exacerbate the issue for testing which is good or a polygon that has transparent edges. Not required though)
3. import render to AE, set to premultiply and choose to premultiply with color BLACK.
4. Composite against a white solid in AE. You should see nothing but white.

Assuming that works, then add Vue to the mix.

1. add some vue content to the render, make it the same, 100% white, but keep the black background.
2. composite the same way.
3. you should see no black halo from the black background you rendered against.
4. keeping the LW polygon and Vue content in differnt parts of the screen will let you see if VUE is messing up or LW is messing up.

Just a thought, I know you said you don't have time to play with it. Just wanted to toss the idea out there.

Captain Obvious
02-04-2010, 02:22 PM
I'm actually starting to wonder if the problem isn't something to do with AE rather than my LW/Vue setup. On this computer sometimes when I work in 32b mode it can act strange.
No Adobe applications function properly with premultiplied footage, so it's no wonder you're having issues. Try using compositing software instead of motion graphics software and I'm sure you'll have greater success. :)

If you can get past the funky UI, Blender is a pretty decent choice.

4dartist
02-04-2010, 02:29 PM
No Adobe applications function properly with premultiplied footage, so it's no wonder you're having issues. Try using compositing software instead of motion graphics software and I'm sure you'll have greater success. :)

If you can get past the funky UI, Blender is a pretty decent choice.

Haha. Photoshop CS3 definitely doesn't recognize premultiplied stuff. But AfterEffects CS3 deals with it just fine. Assuming you set it to interpret the footage that way. Been doing it for years and years, no black halos.

Just out of curiosity what compositor do you use Captain?

clagman
02-04-2010, 02:49 PM
Using CS4 here and same problem with Photoshop (64 or 32 bit). AE is usually fine. Tried this out on the other workstation and...

No problem there. Uh oh. Something going on with mah computer. At least I know I'm not going crazy.

Captain Obvious
02-04-2010, 03:09 PM
Haha. Photoshop CS3 definitely doesn't recognize premultiplied stuff. But AfterEffects CS3 deals with it just fine. Assuming you set it to interpret the footage that way. Been doing it for years and years, no black halos.

Just out of curiosity what compositor do you use Captain?
I was unable to get it to unpremultiply without an external alpha. And how do you get OpenEXR files with an embedded alpha to NOT read the alpha as pixel transparency?

I used Shake up until I started using Fusion, about a year-ish ago.

clagman
02-04-2010, 03:24 PM
I think the alpha is working. I think I am seeing some kind of odd color bleed
around the edges of everything. Doesn't do this on workstation 2. This actually makes my day. All I have to do is figure out what setting on my machine is causing that to look so odd.

Oh but I have to say thanks for the responses though. I friggen love this forum. E-on forum is nowhere near as helpful.

Andyjaggy
02-04-2010, 04:06 PM
Photoshop can load premultiplied alphas if you load it as a video layer, then you have the interpret footage option. :) Just a little trick I learned.

Clagman, I think your problem is Vue. I have never been able to get a good alpha with Vue, I always get the fringing, weather it's premultiplied or a straight alpha.

4dartist
02-04-2010, 04:12 PM
Andyjaggy, are you using CS4 or CS3? If you can do that in CS3 that is pretty awesome.

Andyjaggy
02-04-2010, 04:47 PM
Andyjaggy, are you using CS4 or CS3? If you can do that in CS3 that is pretty awesome.

CS3. One of my co-workers figured it out actually. Ya know the more I use photoshop the more I realize how poorly it does many things. :) I've started doing most tasks in AE, even just for stills. Photoshop is simply too destructive, and too hard to update your composition with new renders.

4dartist
02-04-2010, 04:53 PM
I totally agree with you. Thanks for the tip too.

clagman
02-04-2010, 05:14 PM
Ya Andy that's pretty sweet! I use AE for all my EXR stuff too.

I do believe that I have this alpha situation worked out. I can't say for an absolute 100% until tomorrow morning but the sample exr seemed to composite perfectly on workstation 2. Just not on mine (naturally). Has something to do with 32bpc mode (composite with linear gamma).

Captain Obvious
02-04-2010, 05:28 PM
CS3. One of my co-workers figured it out actually. Ya know the more I use photoshop the more I realize how poorly it does many things. :) I've started doing most tasks in AE, even just for stills. Photoshop is simply too destructive, and too hard to update your composition with new renders.
Yeah, tell me about it. Cityscape does all its stills compositing and grading in Photoshop, and after a project has been running for a while, they spend more time replacing masks and figuring out why the whole thing broke every time there's a new render with new geometry — rather than polishing the thing further.

I'm doing the compositing and grading as well as the lighting for the current project I'm working on, and doing it all in Fusion. Basically, in the time it takes the Photoshop-based retouchers just to put the new render in the right place, I've already replaced *all* the passes with the new renders, just by re-loading the footage, and everything just bloody works.

Photoshop is great for painting. I wouldn't want to be without it for texturing work, for example. But compositing? No thank you! :compbeati




Has something to do with 32bpc mode (composite with linear gamma).
So... is it that it works only when you set it to 'composite with linear gamma' —*or is it that it DOESN'T work if you do? Did you apply any form of gamma adjusting filter in Lightwave or Vue, or is it just handled by Adobe's color profiling of 32bpc footage?

Andyjaggy
02-05-2010, 08:58 AM
Yep photoshop is good for painting, that's about it. I try to keep that opinion to myself cause most graphic designers would fight me to the death over it. They just don't know the beauty of a non destructive and node based workflow.

clagman
02-05-2010, 12:51 PM
I agree AndyJ. It was really nice when they updated Photoshop with non destructive filters but I never use it for video anything.