PDA

View Full Version : exrTrader dilemma



Ma3rk
11-20-2018, 09:18 PM
Wow. Talk about a chasing something down rabbit hole. While trying to render out a frame to show the original problem I'm having, I came across another which, in all honesty, I think is simply my system hitting the memory wall. All the more reason for my rationilza..., justification, for a new, 21st century system in the near future. :D

Anyway, the problem I'm having is this:

I set up exrTrader for a few buffers along with the backdrop. An F9 renders fine and I look at the buffer results and all is as it should be.

This is the "final" (this is a hyper stripped down test scene). My actual scene gets to 99% then hangs if I enable exrTrader. Disabling instances didn't help, so needed something reresentative that I could output. This is using the same enviro light, GI, volumetrics, etc. though.

143391

And the backdrop layer:

143392

But when I look at the resulting exr file layer though, I'm seeing this for the backdrop:

143393

The other layers seem "clipped" too and I've no color info in the object and surface ID layers in the exr looking at them in PhotoShop.

I've had exrTrader for some years but this is the first time I've really used it with 2018, so at a loss what I'm missing here.

Megalodon2.0
11-23-2018, 03:59 AM
Better to post this down in the LightWave 3D Technique Discussions - LW - 3rd party threads.

Here it's too far away from the group that can help you.

Ma3rk
11-23-2018, 08:08 AM
Better to post this down in the LightWave 3D Technique Discussions - LW - 3rd party threads.

Here it's too far away from the group that can help you.

OK, will do. Just figured the holiday was slowing up any responses.

jwiede
11-23-2018, 01:57 PM
OK, will do. Just figured the holiday was slowing up any responses.

Actually, I'd recommend posting over at the DB&W forums, or go on their Discord to discuss it (see their website for details), you're much more likely to get a quick answer there.

jwiede
11-23-2018, 02:01 PM
That's impressively "odd", btw, betting it's some kind of bug just by the looks of it.

Ma3rk
11-23-2018, 03:07 PM
Actually, I'd recommend posting over at the DB&W forums, or go on their Discord to discuss it (see their website for details), you're much more likely to get a quick answer there.

Didn't know they had a Discord. Figured posting here would get more eyes. If nothing after the weekend I'll try over there.

jwiede
11-23-2018, 03:23 PM
Didn't know they had a Discord. Figured posting here would get more eyes. If nothing after the weekend I'll try over there.

It almost looks like as if it's using the foreground as a "clip mask" for where to do proper CS conversion. Very odd, could kind of see how it might be an issue introduced during the recent Cryptomatte changes.

Are you using exrTrader or exrTrader Pro? Any chance you can package and post a test scene? I'd be interested to see if I can repro it here with exrTrader Pro.

Ma3rk
11-23-2018, 05:40 PM
It almost looks like as if it's using the foreground as a "clip mask" for where to do proper CS conversion. Very odd, could kind of see how it might be an issue introduced during the recent Cryptomatte changes.

Are you using exrTrader or exrTrader Pro? Any chance you can package and post a test scene? I'd be interested to see if I can repro it here with exrTrader Pro.

I'm not using the Pro version. Another odd thing is that the UI's Buffer view shows it all correctly; it's just the resulting layer on the OpenEXR file.

Let me see how big the test scene is. It's got an 8K HDR so if not to bad sizewise, I'll post it a bit later.

Lightwolf
11-23-2018, 06:12 PM
Wow. Talk about a chasing something down rabbit hole. While trying to render out a frame to show the original problem I'm having, I came across another which, in all honesty, I think is simply my system hitting the memory wall. All the more reason for my rationilza..., justification, for a new, 21st century system in the near future. :D

Anyway, the problem I'm having is this:

I set up exrTrader for a few buffers along with the backdrop. An F9 renders fine and I look at the buffer results and all is as it should be.

This is the "final" (this is a hyper stripped down test scene). My actual scene gets to 99% then hangs if I enable exrTrader. Disabling instances didn't help, so needed something reresentative that I could output. This is using the same enviro light, GI, volumetrics, etc. though.

I've just been pointed at this thread. Could you get me a sample scene that reproduces the error? You can reach me at [email protected] (or on our discord server).

Also, assuming it's quicker for a start, the EXR image itself would help as well - but I'd need a test scene eventually. From what I've seen I have no idea about what could go wrong.

Cheers,
Mike

Edit: Which application are you using to view the resulting EXR?

Ma3rk
11-23-2018, 07:12 PM
I've just been pointed at this thread. Could you get me a sample scene that reproduces the error? You can reach me at [email protected] (or on our discord server).

Also, assuming it's quicker for a start, the EXR image itself would help as well - but I'd need a test scene eventually. From what I've seen I have no idea about what could go wrong.

Cheers,
Mike

Edit: Which application are you using to view the resulting EXR?

Thanks Michael. I'm trying to prep a stripped down scene now. Even replacing the 8K HDR with a 1K one, the sample scene is still 100 Mb.

I'm sure it's just op error, but I'm at a loss tracking this issue down.

What I'll do instead is post it on WeTransfer & send you the link. I'll post that link here if anyone else wishes to examine. Their links usually are available for 7-10 days.

The only app I have that can look at the layers in the exr is Photoshop or After Effects. I'm still using CS6 extended.

Hmmm. Just threw this into AE applied the Create Pro EXR Layer Comps, and the backdrop is behaving there. The ID layers aren't though. They show properly in Photoshop however.

Anyway here's the resulting layered exr to start with. Had to make it a zip as the forum limits file types.

143406

Ma3rk
11-23-2018, 07:30 PM
For anyone interested, here's the WeTransfer link:

https://we.tl/t-7QzTpKio4p

Will be deleted on 1 December, 2018.

jwiede
11-23-2018, 11:20 PM
Mark, just to clarify: "WindowWork_exp.lws" produces the buffers in question, correct?

I'm not seeing exrTrader used in that scene, all the buffers are PNG, etc. Is there another scene hiding somewhere that I'm not seeing?

Lightwolf
11-24-2018, 03:52 AM
I've just loaded the rendered EXR into Fusion and the saved layers are fine (I had to add some nodes to change the colour space of the colour layers and make the IDs visible).
143407
Affinity Photo also seems to be doing fine:
143408

I haven't looked at your scene yet, that's next. And that 99% hang really is worrying.

Cheers,
Mike

Ma3rk
11-24-2018, 08:56 AM
I've just loaded the rendered EXR into Fusion and the saved layers are fine (I had to add some nodes to change the colour space of the colour layers and make the IDs visible).
143407
Affinity Photo also seems to be doing fine:
143408

I haven't looked at your scene yet, that's next. And that 99% hang really is worrying.

Cheers,
Mike

I'm wondering now if it's something with Photoshop. Might be a reason to pick up Affinity.

I was going to test the hanging issue again. Last night when doing a test, I noticed that the rendering tiles had finished yet progress bar showed only 88%. I waited 30-40 secs and then all completed apperantly; the image widow came up and the exr's had gotten written.

Ma3rk
11-24-2018, 09:00 AM
Mark, just to clarify: "WindowWork_exp.lws" produces the buffers in question, correct?

I'm not seeing exrTrader used in that scene, all the buffers are PNG, etc. Is there another scene hiding somewhere that I'm not seeing?

Hey JW. Go into the Effects Tab under Image Filter and you'll find it. I posted the scene with it simply unclicked so recheck it on. The buffers I enabled will be there but the paths won't be correct obviously.

Lightwolf
11-24-2018, 09:11 AM
I'm wondering now if it's something with Photoshop. Might be a reason to pick up Affinity.
To be fair, Photoshop doesn't have the best OpenEXR support. You should certainly use ProEXR.


I was going to test the hanging issue again. Last night when doing a test, I noticed that the rendering tiles had finished yet progress bar showed only 88%. I waited 30-40 secs and then all completed apperantly; the image widow came up and the exr's had gotten written.
If it was an F9 render, make sure to turn off the Store all Buffers option: https://wiki.db-w.com/exrtrader2018/workflow#store_all_buffers_f9
That may be the cause. Not for the delay itself but for it being excessive.

Cheers,
Mike

Ma3rk
11-24-2018, 12:13 PM
To be fair, Photoshop doesn't have the best OpenEXR support. You should certainly use ProEXR.

If it was an F9 render, make sure to turn off the Store all Buffers option: https://wiki.db-w.com/exrtrader2018/workflow#store_all_buffers_f9
That may be the cause. Not for the delay itself but for it being excessive.

Cheers,
Mike

I'm fairly convinced now that it's a PhotoShop issue. Picked up Affinity Photo this morning; only $39.95 today so not a brain strain to decide. Turned off the Store all Buffers option in exrTrader & F9ed. PShiop still has the problem, Affinity doesn't.

PhotoShop CS6
143412

Affinity Photo
143413

I guess the next question is with the ID layers. In Lightwave with an F9, each item or surfaces displays a unique color. In the output exr layers though, they're more an alpha.

Buffers saved from F9:
143415

But from the same F9, layer from the output exr:
143416

And finally, largely due to your avatar:
143417

I've no idea what it means. The shop was closed & no longer there.

Lightwolf
11-25-2018, 06:25 AM
I guess the next question is with the ID layers. In Lightwave with an F9, each item or surfaces displays a unique color. In the output exr layers though, they're more an alpha.

It's not directly an alpha. What VPR displays is a changed representation of what the buffer contains, but not directly what it contains.
exrTrader on the other hand saves the buffers as LW generates them.

Basically, the ID buffers in LW store a numeric value per pixel which represents either the most used or last rendered surface/item for that pixel (to be fair, I don't know which strategy LW uses).
So, a range of numbers start at 0 and proceeding from there: 1,2,3,4 etc...

Now, if you know how HDR images are interpreted you'll see that the number are essentially outside of the visible range.

So, unless your package can deal with the numeric values directly (i.e. Fusion) you'll need to change the exposure to bring the IDs in a visible range and then, essentially, luma key them.

I am thinking about generating "clown" mattes in exrTrader (based on the work for Cryptomatte), but that would be a Pro feature only (exrTrader saves all the buffers as generated by LW, Pro also generates its own buffers - that is the main difference. Cryptomattes being the first implementation for Pro).



And finally, largely due to your avatar:
143417

I've no idea what it means. The shop was closed & no longer there.
I have no idea either, but you can never have enough parts. ;)

Cheers,
Mike

Ma3rk
11-27-2018, 12:36 PM
It's not directly an alpha. What VPR displays is a changed representation of what the buffer contains, but not directly what it contains.
exrTrader on the other hand saves the buffers as LW generates them.

Basically, the ID buffers in LW store a numeric value per pixel which represents either the most used or last rendered surface/item for that pixel (to be fair, I don't know which strategy LW uses).
So, a range of numbers start at 0 and proceeding from there: 1,2,3,4 etc...

Now, if you know how HDR images are interpreted you'll see that the number are essentially outside of the visible range.

So, unless your package can deal with the numeric values directly (i.e. Fusion) you'll need to change the exposure to bring the IDs in a visible range and then, essentially, luma key them.

I am thinking about generating "clown" mattes in exrTrader (based on the work for Cryptomatte), but that would be a Pro feature only (exrTrader saves all the buffers as generated by LW, Pro also generates its own buffers - that is the main difference. Cryptomattes being the first implementation for Pro).Mike

I think colors would be easier to work with certainly if you've dozens of objects or surfaces. With the numeric route, it's more of a guessing game.

I should make another sortie into learning Fusion. Haven't largely due to the vast number of items I've gathered over the years for After Effects I guess. For simple comping tasks, it might be worth getting comfortable with.




I have no idea either, but you can never have enough parts. ;)

Cheers,
Mike

If ya have the room to keep 'em, I suppose. I'm still a bit confused as to what might be wearing out.

M.

Ma3rk
11-27-2018, 09:27 PM
Hey Mike. Tried that scene again where Trader hung. It did again, but I'm pretty sure it's simply a system mem issue. Taskman showed it pegged whereas CPU use was 1%.

I let it percolate for another 10 mins or so after the F9 render hit 100%, which was in the neighborhood of 22 mins.

Lightwolf
11-28-2018, 02:59 AM
Hey Mike. Tried that scene again where Trader hung. It did again, but I'm pretty sure it's simply a system mem issue. Taskman showed it pegged whereas CPU use was 1%.

I let it percolate for another 10 mins or so after the F9 render hit 100%, which was in the neighborhood of 22 mins.

Memory would explain it. exrTrader tries to be conservative when getting the buffers from LW, it's only one scan line at a time. But then the buffers themselves need memory. This is where turning off the "Store all Buffers" option, which you did, helps.

Cheers,
Mike