PDA

View Full Version : Suspecting bug in clearcoat



Dan Ritchie
12-05-2018, 10:43 AM
I have 2 images. One has clearcoat all the way off. The other had clearcoat at 1 percent. That's the only change between the 2 images. The first has minimal to no fireflies. The second has a big problem that aa is unable to mitigate.
At one percent, I would expect any fireflies from clearcoat to resolve out after a few passes, but here's 400 aa passes, and they only get worse and worse with aa passes.


143488

143489

lertola2
12-05-2018, 01:35 PM
If you turn on Limit Dynamic Range on the Effects - Processing tab does that help?

Dan Ritchie
12-05-2018, 01:41 PM
Yes, but that doesn't keep it from being a bug.

Dan Ritchie
12-05-2018, 01:54 PM
After some more testing, I'm convinced this is a clearcoat bug. Perhaps clearcoat is not energy preserving. Anyone else having this problem.
Do I need to provide a scene, or is this clear enough?

Dan Ritchie
12-05-2018, 02:30 PM
Here's a scene demonstrating the problem
http://squirreldome.com/Download/LWO.zip

Dan Ritchie
12-05-2018, 02:43 PM
scene after 1000 aa passes143491

jwiede
12-05-2018, 03:04 PM
It definitely looks like an issue. There seem to be significant issues regarding interactions where even small pBSDF "Roughness" term values combined with any specular/reflective-type term values yield "fireflies" in quantity. There may also be an AA issue, as it seems to identify these "fireflies" as fine details meriting preservation or similar, instead of trying to mitigate their contrast.

rustythe1
12-05-2018, 03:07 PM
Jason posted on FB that the problem goes away if you enable mipmapping, however not sure you are using a BG image, which I would say is usually the main culprit for creating fireflies in gloss/conductor

lertola2
12-05-2018, 07:49 PM
I think the fire flys are some kind of reflection artifact. I turned off the light and set a gray background and set roughness to zero. Interestingly the fire flys only show up where the object self reflects. There is nothing in the scene that could yield a bright reflection as far as I can see. Turning on limit dynamic range gets rid of the fire flys. I think that means that the fire flys are so bright that antialiasing cant correct them. So I hope this is a bug that can be fixed. For the moment it seems like we should not use clear coat and metallic setting together unless we are willing to turn on limited dynamic range.

https://forums.newtek.com/attachment.php?attachmentid=143492&d=1544062041

omichon
12-06-2018, 01:26 AM
After some more testing, I'm convinced this is a clearcoat bug. Perhaps clearcoat is not energy preserving. Anyone else having this problem.


Same conclusion here too. Most of my fireflies issues are gone since I avoid using clearcoat.

Ma3rk
12-06-2018, 08:51 AM
Uncheck the Glossy Reflections option for the surface. You have that enabled plus Affect Specular for the light. You'll need to tweek other surface settings but gloss & sheen will behave more predictably, but that'll get rid of the fireflys.

You should probably set the scene up with an enviro sphere & enviro light rather than just a distant to test correctly.

Dan Ritchie
12-06-2018, 11:41 AM
There's fireflies, and there's incorrect fireflies, and that seems to be what we have here with clearcoat.

lertola2
12-06-2018, 08:08 PM
I put in a bug report for this yesterday. Today I got a message from Lightwave Customer Support: "This has been fixed internally. This should not be reproducible in the next available version."

Yes!

David_CGC
12-06-2018, 09:55 PM
It definitely looks like an issue. There seem to be significant issues regarding interactions where even small pBSDF "Roughness" term values combined with any specular/reflective-type term values yield "fireflies" in quantity. There may also be an AA issue, as it seems to identify these "fireflies" as fine details meriting preservation or similar, instead of trying to mitigate their contrast.

Admittedly, I'm not a programmer, but I don't understand why the renderer can't have a sanity-check of some sort. If it gets a result on a single pass (or even every pass!) of a single pixel that's a hundred or a thousand times brighter than the other passes, and the surrounding pixels, it's far more likely that the value is an error than that there's one impossibly-bright object in the scene that's smaller than a single pixel in the render (well, hundreds of such objects scattered around the frame). I basically do it manually to clean up fireflies in my stills, cranking down the exposure of an exr render until the fireflies are all that's left, and then using that as a mask and using content-aware fill on my main layer.

Ma3rk
12-06-2018, 11:58 PM
Mmmm, well, just sayin'...

143500

143501

RPSchmidt
12-07-2018, 06:52 AM
Okay... so, while I know a bug report has been submitted and that Newtek confirmed that it was a bug, I just wanted to add my experiences / interpretation of clearcoat for reference. Perhaps it will help people who want to work with it, but so far have not found themselves able to because of fireflies.

In my experiences with clearcoat, I have come to believe that the percentage for it is not the amount per se... it's the coverage of the object by the clearcoat.

So, when I use clearcoat, I set it for 100% (full coverage). The specular setting (again, in my experience) is really what controls how strong the effect is.

Using that logic, if you set clearcoat for 1%, you are covering only a small area of each polygon at angle of the object... and using a high specular with that very small area causes blow out. That's where the fireflies come from.

Try this experiment.

Using the dragon, preferably with an HDRI textured environment and environment light set at 1.0 lx (no other light source) set the clearcoat for 100% and the specular for 1%.

You should get a beautiful render in just a few minutes using relatively low render settings.

These are the render settings I used (sorry, wish I could post an image but I can't);

RENDER TAB

All boxes checked at the top (Raytrace Shadows, Reflection, Refraction, etc.)

Diffuse Bounces 5
Ray Recursion 6
Transparency Recursion 6
Reflection Recursion 4
Refraction Recursion 4
Ray Precision 10
Reflection Samples 10
Refraction Samples 10
Subsurface 1

(Global MipMap, Flare, Light, Edge are at the default of 1)

Enable Noise Filtering checked
Render Tile Size 8

GLOBAL ILLUMINATION

Monte Carlo - Brute Force

Intensity 100%
Rays 1

Caustics, Sample Backdrop, ISBG Sampling checked

ISBG Samples 1024

CAMERA SETTINGS

Here, I think the only setting that might be important is Samples.

I turned off Adaptive Sampling and set the Minimum Samples to 15 (although I think you could still get great results with minimum samples of 10).

No Motion Blur or Particle Blur for this one, and no DOF.


The render takes about six minutes on a dual Xeon system at 1920x1080. For faster renders, I'm sure the settings can be played around with to get a good mix of speed and quality.

Give it a try using these settings and let me know what you think.

Dan Ritchie
12-07-2018, 09:54 AM
Admittedly, I'm not a programmer, but I don't understand why the renderer can't have a sanity-check of some sort. If it gets a result on a single pass (or even every pass!) of a single pixel that's a hundred or a thousand times brighter than the other passes, and the surrounding pixels, it's far more likely that the value is an error than that there's one impossibly-bright object in the scene that's smaller than a single pixel in the render (well, hundreds of such objects scattered around the frame). I basically do it manually to clean up fireflies in my stills, cranking down the exposure of an exr render until the fireflies are all that's left, and then using that as a mask and using content-aware fill on my main layer.

Ridiculously bright spots are a valid part of the rendering equation. You see them when the sun glistens off a tiny drop of water, or some glitter, or a bit of sand, or just about anything. Ideally, given a near infinite number of aa passes/samples, it should resolve into some kind of picture. Unless, of course, the bright spots are errors, like these.

Dan Ritchie
12-07-2018, 10:03 AM
I suspect clearcoat is using a non-physically plausible specular type.

MonroePoteet
12-07-2018, 02:42 PM
I've mentioned this in a couple of other threads about fireflies, but the idea doesn't seem to "catch on". IMO, this is a day-one bug in LW2018 in that the Intensity of the Light's reflections are not attenuated with distance as is their illumination. So, even a low-intensity Light at quite a distance from a specular (or in this thread, metallic and clear-coated) Surface will produce "hot spots" on flat facets of the surface.

Attached is a sample scene to demonstrate the "problem" which contains:


A 50% Specular unsmoothed blue sphere with flat facets
A Distant Light, 0.1 lux, about 283 meters from the object
A Point Light, 3.141 lux, almost 40 meters from the object
An Area Light, 3.141 lux, shrunk down to (.1,.1,.1), over 40 meters away from the object


Each of the Lights was carefully positioned to reflect in one of the facets of the unsmoothed sphere.

143504

With GI off and a plain gray background, as can be seen in the render none of the Lights have sufficent illumination to show any color at all on the object, but all three of them show "hot spots" as reflected in the Specular surface:

143503

The Distant light is a LONG way away from the object, but still produces an extreme hotspot in one of the facets of the sphere.

I postulate that when these inadequately attenuated hot spots are reflected (or re-reflected) in the micro-facets used to implement Roughness, it produces the fireflies. IMO, the reflection of a Light should attenuate with distance at approximately the same rate as the Light's illumination intensity. Personally, I would rather have the reflection of a Light be able to be completely disabled by clearing the Visible to Camera flag in the Light's Properties.

Hopefully, the bug submitted by lertola2 in a previous post and "fixed internally" will have fixed this Light Reflection problem not just for Clearcoat, but for reflective surfaces in general (including Specular and Metallic).

mTp

Dan Ritchie
12-07-2018, 03:27 PM
I suspect you would have to have a separate render buffer for every light.

jwiede
12-07-2018, 03:34 PM
Unfortunately, as they've apparently "moved on" from LW2018 development, that apparently means said fix (w.r.t. a serious problem present since LW2018's initial release) will conveniently (for Newtek, not LW2018 customers) arrive in a for-pay version upgrade.

I really hope they do the right thing and release the fix for LW2018, given how long we've been complaining about firefly problems, and how significant a defect they represent (neither limiting LW to LDR rendering nor turning off "glossy reflections" represent broadly-acceptable solutions or even workarounds, they're both too limiting). Whether they've "moved on" or not, it's a significant, broadly-impacting defect in the current product, and as such, merits a fix for the current product.

jwiede
12-07-2018, 05:03 PM
Ridiculously bright spots are a valid part of the rendering equation. You see them when the sun glistens off a tiny drop of water, or some glitter, or a bit of sand, or just about anything. Ideally, given a near infinite number of aa passes/samples, it should resolve into some kind of picture. Unless, of course, the bright spots are errors, like these.

When the sun or similarly bright source glistens off invisibly-micro-scale reflectors, though, it doesn't yield a hyper-bright long-distance-visible reflection, and thus the problem. These fireflies are both much too bright and much too large (esp. relative to roughness micro-facets) to be reasonable representations of such micro-scale reflections. It's an energy-conservation problem: These are much, much brighter (and larger) reflections than their responsible sources should be able to produce given size and distance of camera, even with full sun as source.

Just a SWAG, but what we're seeing is almost as if all the micro-facets representing a given pixel were all (or even most) reflecting in the precise same direction, that or somehow the energy return is being wildly magnified for each micro-facet. Also, that it seems to happen even when roughness is set to 0% implies either there's some roughness being added* (in certain cases, esp. self-reflection) even when set to 0% or something similar is occurring. Having higher roughness values obviously worsens the effect. Regardless, it's a problem, and certain specular settings appear to exacerbate it significantly.


*: To clarify: By itself, that's not unreasonable. There's really no such thing as "material roughness = 0%" at the scales we're discussing. However, in combination with whatever else is occurring, it makes the issue difficult to avoid.

lertola2
12-07-2018, 05:26 PM
I've mentioned this in a couple of other threads about fireflies, but the idea doesn't seem to "catch on". IMO, this is a day-one bug in LW2018 in that the Intensity of the Light's reflections are not attenuated with distance as is their illumination. So, even a low-intensity Light at quite a distance from a specular (or in this thread, metallic and clear-coated) Surface will produce "hot spots" on flat facets of the surface.

Attached is a sample scene to demonstrate the "problem" which contains:


A 50% Specular unsmoothed blue sphere with flat facets
A Distant Light, 0.1 lux, about 283 meters from the object
A Point Light, 3.141 lux, almost 40 meters from the object
An Area Light, 3.141 lux, shrunk down to (.1,.1,.1), over 40 meters away from the object


Each of the Lights was carefully positioned to reflect in one of the facets of the unsmoothed sphere.

143504

With GI off and a plain gray background, as can be seen in the render none of the Lights have sufficent illumination to show any color at all on the object, but all three of them show "hot spots" as reflected in the Specular surface:

143503

The Distant light is a LONG way away from the object, but still produces an extreme hotspot in one of the facets of the sphere.

I postulate that when these inadequately attenuated hot spots are reflected (or re-reflected) in the micro-facets used to implement Roughness, it produces the fireflies. IMO, the reflection of a Light should attenuate with distance at approximately the same rate as the Light's illumination intensity. Personally, I would rather have the reflection of a Light be able to be completely disabled by clearing the Visible to Camera flag in the Light's Properties.

Hopefully, the bug submitted by lertola2 in a previous post and "fixed internally" will have fixed this Light Reflection problem not just for Clearcoat, but for reflective surfaces in general (including Specular and Metallic).

mTp

I am not sure if this scene points out a bug or not. Are you trying to demonstrate that lights so distant from the an object should be so dim that they should not be visible in reflections? All the rays from a distant light are parallel so it makes not difference where a distant light is placed. Only its rotation matters so its not surprising that a distant light is visible in the reflection. For the area light you have normalize turned on. I was watching RebelHill's tutorial on lighting last night and he pointed out that with normalize turned on for area lights then they will give out the same amount of light no matter what size they are. So a very small area light with normalize turned on should be the same as a point light. If you turn normalize off then the area light is not visible in the reflection which is how it should be I think. So for the point and area light I take it that you think the reflection is too bright given how far they are from the sphere? But how can we tell how bright they should actually be?

I agree that it would be good if turning off visible to camera would also make lights invisible in reflections.

RPSchmidt
12-07-2018, 08:35 PM
I am not sure if this scene points out a bug or not. Are you trying to demonstrate that lights so distant from the an object should be so dim that they should not be visible in reflections? All the rays from a distant light are parallel so it makes not difference where a distant light is placed. Only its rotation matters so its not surprising that a distant light is visible in the reflection. For the area light you have normalize turned on. I was watching RebelHill's tutorial on lighting last night and he pointed out that with normalize turned on for area lights then they will give out the same amount of light no matter what size they are. So a very small area light with normalize turned on should be the same as a point light. If you turn normalize off then the area light is not visible in the reflection which is how it should be I think. So for the point and area light I take it that you think the reflection is too bright given how far they are from the sphere? But how can we tell how bright they should actually be?

I agree that it would be good if turning off visible to camera would also make lights invisible in reflections.

Also, did you set up a falloff node for any of them? If not, aren't they all essentially unlimited light, i.e., their intensity doesn't fall off with distance at all?

lertola2
12-07-2018, 08:52 PM
Also, did you set up a falloff node for any of them? If not, aren't they all essentially unlimited light, i.e., their intensity doesn't fall off with distance at all?

The lights do have inverse squared falloff set (except for the distant light of course) so they don't have to have nodal falloff I think.

gdkeast
12-08-2018, 12:00 AM
Yes, I totally agree. I would hope known and documented bugs are fixed in 2018 before "onto the next."

MonroePoteet
12-08-2018, 08:00 AM
Well, as I said, it's only a hypothesis. To me it makes no sense that a 0.1 lux DistantLight (with 0.0 Angle) almost 300 meters away from an object which does not illuminate that object at all (or just barely) would show a huge, bright reflection in it. To me it makes no sense that a POINT Light would show any sizeable reflection at all.

If I replace the Area Light with a luminescent sphere of the same size and same distance from the faceted, specular sphere, I have to increase the luminosity of it to over 2000% for it to be visible in the reflection at all, and then it appears as a tiny, indistinct dot. To me it makes no sense that a 3.141 lux Area Light which doesn't illuminate the faceted sphere at all would have a bright, distinct reflection on that object.

In conclusion, I hope LWDG can provide some relief from the fireflys, whatever their source may be!

mTp

Dan Ritchie
12-08-2018, 02:41 PM
To me it makes no sense that a 0.1 lux DistantLight

Agreed with that. Unfortunately, for direct lighting it's still more practical to use some fakery to get things done. If somebody knows that there's an accepted solution to this, please speak up.

kyuzo
12-08-2018, 05:04 PM
Unfortunately, as they've apparently "moved on" from LW2018 development, that apparently means said fix (w.r.t. a serious problem present since LW2018's initial release) will conveniently (for Newtek, not LW2018 customers) arrive in a for-pay version upgrade

I've been out of touch because of work upheavals... What have I missed with regards to development announcements?

jwiede
01-22-2019, 07:12 PM
Well, as I said, it's only a hypothesis. To me it makes no sense that a 0.1 lux DistantLight (with 0.0 Angle) almost 300 meters away from an object which does not illuminate that object at all (or just barely) would show a huge, bright reflection in it. To me it makes no sense that a POINT Light would show any sizeable reflection at all.

If I replace the Area Light with a luminescent sphere of the same size and same distance from the faceted, specular sphere, I have to increase the luminosity of it to over 2000% for it to be visible in the reflection at all, and then it appears as a tiny, indistinct dot. To me it makes no sense that a 3.141 lux Area Light which doesn't illuminate the faceted sphere at all would have a bright, distinct reflection on that object.

I agree Monroe, it definitely doesn't make sense for such attenuated lights to generate micro-hotspots the way they are, when their general illumination contributions are so faint. The renderer is almost behaving as though falloff isn't properly being considered when it comes to specular highlights, nor micro-facet-related glints (from roughness).

Unfortunately, the scene appears much the same in LW2019.0.0 (tested on Mac64, but I expect Win to behave similarly in this case) as it did in LW2018.0.7.

Despike and denoising are great and all, but ultimately, they're just not going to magically solve all the issues involved. If this is all because the renderer isn't properly attenuating certain highlights (as it appears), this means there are legitimate "middle ground" scenarios where some such highlights should exist, but instead wind up coming out nastily over-bright/-large. In those cases, just filtering the highlights (as spikes/noise) actually reduces the quality / accuracy of the result, rather than improving it.

jbrookes
01-25-2019, 12:18 PM
I'm having similar issues with Principled and HDR Environment images.

I found that dropping Specularity all the way down to between 2 and 2.5 helps (along with Roughness at zero and a Clearcoat setting of 3).

Light Samples seem to help smooth out the hot spots (up to a light sample setting of around 48), but don't seem to help with the fireflies that crawl all over the surface during animations.

There's no question that both Specular and Clearcoat are hyper-sensitive and blow out really easily. The above Spec and Clearcoat settings seem to help quite a bit with both the light source hot spots and the speckles/fireflies.

Not at my LW workstation right now, but I'm pretty sure I have Subsurface at zero.

jbrookes
01-25-2019, 10:21 PM
Okay, I checked my settings and it turns out that my Subsurface is set to 5%.

I tried raising it to 100% -- like for your settings -- and thinks got noisy really fast. Even 50% is pretty wild.

So perhaps try something like:

Roughness = zero
Specular = 2.5%
Spec Tint = 0
Tint = 0
Translucency = 0
Flatness = 0
Subsurface = 5%
Distance = 10 cm (it's a 1m sphere in my case)
Luminous = 0
Metallic = 0
Anisotropic = 0
Clearcoat = 3%

jbrookes
01-25-2019, 11:56 PM
Last post from me for the night.

Here's a couple that worked quite well for me:

This one's sort of a semi-gloss (matte-ish) paint job:
Roughness = 10
Specular = 2.5%
Spec Tint = 0
Tint = 0
Translucency = 0
Flatness = 0
Subsurface = 5%
Distance = 1m
Luminous = 0
Metallic = 80
Anisotropic = 0
Clearcoat = 3%

This second example is more of a glossy paint job:
Roughness = 3
Specular = 2.5%
Spec Tint = 0
Tint = 0
Translucency = 0
Flatness = 0
Subsurface = 5%
Distance = 1m
Luminous = 0
Metallic = 10
Anisotropic = 0
Clearcoat = 3%

lardbros
01-26-2019, 02:35 AM
What are your SSS samples set to?
The higher your SSS, the more samples you'll need too.

jbrookes
01-26-2019, 06:17 PM
So I took a look at this.

lertola2/RebelHill's explanation covers the Area light. Indeed, unchecking Normalize appears to solve that one.

I noted that the point light's reflection does decrease as the point light is moved farther away from the object. Whether it's decreasing as much as it should is up for debate.

And as mentioned, the distance of the Distant Light won't make any difference. I did try playing around with the Distant Light's intensity and I got a reasonable reflection at around 0.001 to 0.005 lx. However, it isn't until the Distant Light it upped to 0.5 lx that you get enough illumination to match the illumination of the object's surface and the associated hotspot so that, together, they look realistic (IMHO). So I see what MonroePoteet is saying (at least for distant lights anyway). I suppose a workaround would be to twin up the Distant Light and have one that handles light reflections and another that handles diffuse surface illumination. The other option is to maybe avoid using distant lights?

Anyone feel like doing a study at sunrise or of a sun setting over a mountain (armed with a mirror and a diffuse flat material) to see what the ratio of diffuse illumination to light reflection hotspot should be in real life?

Bill Carey
01-27-2019, 06:30 AM
Completely off topic. Is there a place to get these ‘standard’ models i see used everywhere? This dragon, Ajax, David etc. would be nice to have the same references.

lardbros
01-27-2019, 02:28 PM
You can get some nice scanned models from here, statues, crabs and things:
https://www.turbosquid.com/Search/Artists/oliverlaric

The Ajax bust model can be found here:
https://www.yobi3d.com/q/3D-model/ajax-jotero-com

jbrookes
01-29-2019, 02:14 PM
I know the dragon's included with LW2018...