View Full Version : Benchmark marbles / quality of render cross check?

08-13-2015, 04:41 AM

some of you might know about the marbles benchmark thread:

benchmark marbles thread (http://forums.newtek.com/showthread.php?133251-11-5-s-BenchmarkMarbles-lws-share-your-machine-s-render-time-here&highlight=benchmark)

For fun I just ran it on Win10 Pro with a fresh install of LW 2015.3 (importance sampling = off to be comparable to LW 11.x). Got my usual 37 / 38 minutes (dual Xeon machine). But watching the image instead of the figures I noticed that the white marble is quite noisy and the highlights are blown out (I think it was the same with the other 2015 versions).


The screenshots of older LW versions in the mentioend thread look cleaner. I don't have LW 11.x anymore on my pc. Does anybody have saved an original render made with an older LW to compare (preferably as png so that jpg compression does not fool me here)?

10-07-2015, 10:57 PM
I've been experimenting with Global Illumination in LW 2015.3, and in particular, how Importance Sampling affects both render quality and render time. Here are three renders of the Benchmark Marbles scene, done on a 2013 Mac Pro 6-core with 32 GB RAM, AMD FirePro D500 3072 MB, running OSX 10.10.5, with all typical services running (internet, email, etc.) -- In all cases, the scene was simply opened and F9 was hit, except Render 2, with Importance Sampling manually turned off.

Render 1: LW 2015.3, with Importance Sampling ON, since it is on by default. Render time: 1:58:24


Render 2: LW 2015.3, with Importance Sampling OFF. Render time: 1:31:51 -- Wow, that's 26:33 faster.


Render 3: LW 11.6.3, which doesn't have Importance Sampling. Render time: 1:25:36 -- 32:48 faster than LW 2015.3 with Importance Sampling ON, and 6:15 faster than 2015.3 with Importance Sampling OFF.


One might conclude that Importance Sampling causes render times to be SLOWER. -- ?? (Page 2198 of the 2015 Update 3 manual says, "Importance Sampled BG - provides you with a way to use GI settings that provide a quicker
render, but still have a smooth background in the image."

Regarding image quality, there are difference between the three renders, but they are minor. Importance Sampling does deliver a slightly "smoother" result. Render times, though, are surprising, given the statement in the 2015.3 docs that IS would make them faster. In my tests, between 2015.3 and 11.6.3, this is not the case.

10-08-2015, 12:53 AM
Well... a lot of things have changed since that scene was created. 11.5 is not 11.6.3 is not 2015.3. If you look closely, you'll notice that many things are not the same in your renders compared to the original that Dave and Cellshader posted. Flip back and foth between them all. There's other stuff going on, so it isn't an apples/apples test. Essentially, that scene needs to be updated and the thread re-started using current tech.

10-08-2015, 01:05 AM
What would you change in the benchmark scene?

10-08-2015, 11:37 AM
Importance sampling didn't exist when that scene was created.

10-08-2015, 12:49 PM
After some reflection on my render tests in 11.6.3 and 2015.3, it is true that using Importance Sampling does take longer to render with the benchmark 11.5 scene (or any other scene with the same Render, GI, and Camera settings), but ...

The result using Importance Sampling is less noisy. Yes, rendering does take longer than NOT using Importance Sampling, but the final render happens faster than achieving the same lower noise levels WITHOUT Importance Sampling. This is why the docs claiming faster renders with Importance Sampling is actually true.

Here are three more renders to compare; they all use the same Render, GI, and Camera settings as the original 11.5 Marbles Benchmark Scene. I used a ground poly with no texture except a light gray color, to focus on NOISE.

1. LW 11.6.3 - Render time = 55:50

2. LW 2015.3 - Importance Sampling OFF - Render time = 56:36

3. LW 2015.3 - Importance Sampling ON - Render time = 1:27:12

If you pre-load them all, you can switch between them quickly. I see NO difference between 11.6.3 and 2015.3 with Importance Sampling OFF. 2015.3 with Importance Sampling ON is less noise, but of course, takes longer to render.

10-08-2015, 07:37 PM
2015.3 with Importance Sampling ON is less noise, but of course, takes longer to render.

And worth it!

10-11-2015, 01:09 AM

may I ask whether you can post your scene here?
I like to test how would the non importance render look if it would run the same time like the importance one.

As in:
55 or 56 mins non importance = noisy
87 mins with importance = less noisy
87 mins non importance = ?

And as I understand it, importance sampling only has an effect if I use the textured environment, right?
If I put an env-sphere around my scene and e.g. texture it with an hdr, importance sampling won't help, right?

10-11-2015, 08:02 PM
The result using Importance Sampling is less noisy. Yes, rendering does take longer than NOT using Importance Sampling, but the final render happens faster than achieving the same lower noise levels WITHOUT Importance Sampling.

Please post your scene so we can test further?

I'm not convinced that 11.6.3's render taking the same amount of time as 2015.3's render would have significantly more noise, the noise improvement shown is pretty close to what I'd expect to achieve by setting 11.6.3 for time-equivalent rendering.

The problem here seems to be a lack of clarity as to precisely how Importance Sampling (IS) is supposed to function in LW. In other packages, enabling IS causes the renderer to distribute the same general number of samples but using the IS algorithm to do so, thus time remains largely the same but quality is better for time. In LW, clearly it isn't holding the general number of samples constant and just adjusting their distribution, because if it were, time required shouldn't increase so substantially (unless bugged).

So the question becomes: When IS is enabled in LW, what happens w.r.t.:

1. The overall number of samples?
2. How the IS algorithm allocate more/less samples?
3. Is there any way to track/log the adjustments the IS algo makes?

Given the expected function of the IS algo, having it function as it is (causing greater time cost) is very sub-optimal from a tuning perspective, IMO, because the additional time required is essentially unpredictable. Predictability of render time cost is too important to tuning and optimization, and holding samples constant but redistributing per IS algo achieves that, while the current approach does not retain predictability.

10-12-2015, 05:07 PM
Here you go ... pretty simple. 130316

Jweide, I had similar thoughts, once I did some testing, with the 11.5 benchmark scene, and the simple scene I made. My eyes are getting pretty old, but I see virtually NO difference between the 11.6.3 render and the 2015.3 renders. The real difference is in the 2015.3 render with Importance Sampling turned on: much less noise. But I agree; since it takes longer to achieve the better result, it would be nice to know how much longer renders would take without IS.

For me, IS is nice, but so far, it's a bit of a "black box."