PDA

View Full Version : Major scene loading performance issues?



Hieron
12-10-2012, 10:29 AM
I'm a tad flabbergasted, and unsure why this is even happening.. if anyone knows a way to turn this behavior off, please let me know.. this has cost me so much time already it ain't funny. No clue why this doesn't come up more often.

2 issues:
-Loading a scene with 200 clones of a single object file has Layout loading the file 200x. Why? ( I can live with this, still weird though)
-Loading a scene with 200 objects with sort of doable nodal displacement trees is insanely slow. Layout is loading object 1, parsing whole scene displacement? Takes 0.01 secondsas the object is the first. No problem! Loads object 2, parses whole scene, takes 0.02 seconds, no problem. We are at 0.03 seconds total now. But see where this is going? My full scene is 800 objects which would take an hour + to load in Layout even though a single parse is seconds. Amazing Denis graciously helped and turned off parsing of his nodes which made this even possible, otherwise it would be even 2x slower. Astonishingly, lwsn.exe doesn't do this behavior and loads the scene in seconds as it does not keep parsing before all is loaded.

Please tell me I missed some obvious switch, commandline flag for Layout or something...

If not, could stuff like this please be considered for future versions.. Multithreading as well? I thought scene loading was improved in 10 or 11..

Hieron
12-11-2012, 07:06 AM
right another one, a workaround to dodge above issues:

-adding ClothFX to all
-Scaning motion, saving all (nice how this works, quite easy)
-Delete nodal displ. connections by Python Script
-Loads fast!

but introducing a new:
-Layout gets semi stuck in endless refreshing calls making the UI quite unusable.

So the loading speeds is fixed but traded vs hardly anu UI response... pfff... some days..


ps: this doesn't happen on smallish scenes, but there is no obvious way why it would happen on this 1000 cell scene.

Phil
12-11-2012, 08:05 AM
Report it, with content. It's the only way NT will be able to wrangle it into submission.

Ryan Roye
12-11-2012, 08:12 AM
I ran into this issue too for a larger scene I'm doing. I ended up having to use Dpont's MDD pointer instead of Clothfx to load up the large volume of MDD animations I had. Clothfx driven MDD also doesn't seem to allocate memory efficiently, I had a lot of memory ceiling issues even though I was using 1 MDD to drive the animation of a hundred or so low-poly models.

Hieron
12-11-2012, 09:19 AM
interesting. Do you know of a way to quickly swap out clothfx driven .mdd to DP's MDD pointer? I suppose that would need some serious scripting magic?
(got 1000 cells)

Hieron
12-13-2012, 04:43 AM
So, no one ever noticed Layout slowing down to a totally unnecessary crawl when loading in bigger scenes with many objects due to a full and unskippable refreshing after each separate object is loaded? Will come up with simple sample scenes to add to this post and send in to NT in a few hours then..

Phil
12-13-2012, 05:28 AM
I've never seen it. I just wonder if something like FSPE is enabled in your case. I'm interested in this issue as well, but haven't had time to investigate - LScript has been taking all of my spare time of late.

Hieron
12-13-2012, 05:46 AM
Nope no FSPE enabled.. tried things like disabling viewports etc too. Stuff like that helps in Maya, to disable the UI to keep it from refreshing etc. (even though visually nothing is seen to refresh)

I guess that most people do not try to load a 1000 objects with nodal displacements often. Yet it is quite workable fast in Layout and it loads/renders fast on lwsn.exe. The entire problem is the loading into Layout.. What may take Layout an hour, would be seconds in lwsn.exe, surely that would be something that could be remedied?

But this should be a simple problem/case for the devs, not needing any content:
-Does Layout "Load Scene" (and similar) refresh per object loaded?

If yes, then the slow loading times of these scenes is just a straightforward conclusion. But one would wonder why it would be refreshed.. to allow the abort? No idea...

Hieron
12-19-2012, 04:54 AM
Please see object and scenes included below.

CellSpline_setup.lws
The basic setup. 2 cloned objects. Cell (2) parented to Cell (1) and using nodal displacement to follow a spline. (DPKit required, it uses its Spline deformer and Item ID nodes)

CellSpline_200cells.lws
Same scene as before, but with Cell (2) cloned 200x. Done by cloning 5x and making a small parent tree, then using "clone hierarchy" and parenting each time. Do note that Layout will crash when trying to clone too many at the same time. Somewhere between 100 and 150. Scene needs to be saved, Layout closed, reopened and then continued to reach 200.
Takes 42 seconds to load, taking +-0.6 seconds per object at the end.

-> bug/issue: why does it need to load the very same 2kb file each and every time again? Wouldn't it be a time saver if this would be a tad smarter? However this is not the big issue at hand, that is the next point.
-> bug/issue: why does it seem to do a refresh/parse per object loaded, making loading times go ever slower?


CellSpline_400cells.lws
Scene is CellsSpline_200cells.lws x2 by using "load items from scene" on itself to generate double the numbers.
Takes 5.5 minutes to load, taking +- 2.5 seconds per object at the end. So doubling the numbers, makes Layout loading about 8x slower.

!However: loads in about 5 seconds on the renderfarm with lwsn.exe, or about 66x faster than Layout.

->bug/issue: the issue becomes ever more prominent with loading times growing extremely fast and these kinds of numbers are not low for our usage scenarios, we need about 1000-2000 which quickly makes loading taking hours... UI responsiveness when loaded is doable.

Clearing scene/loading from scene/cloning items with such scenes is terribly slow.

->bug/issue: why would it take long to clear the program? Killing the process and restarting can be 50x faster than waiting for it to clear, ever so slowly. This looks like some clearing script running, being terribly slow.

It is completely prohibiting the use of LW on some projects for us.. due to it having issues with loading a 2kb file many times and just waiting to parse the scene until fully loaded.. feels silly and unnecessary.

109973

Hieron
12-19-2012, 05:39 AM
See object and scenes in the package provided for this issue concerning many ClothFX .mdd'ed objects

A work around for the issues presented above (Layout having problems with many nodally displaced objects in a scene):
-apply ClothFX to all objects (by script, easy)
-press "calculate" and abort right away, step needed to not have a crash in the next step
-scan motion
-in FX property, press "save" and it will request a save file for each object in the scene (just keep "enter" pressed down, no sweat)

-> easy!

See CellSpline_400cells_clothFXed.lws for the result. The displacement is cached for just frame 0-2 to keep .mdd size small in this example.

-> bug/issue: even though displacement is now cached, Layout will still parse the nodal displacements and be just as slow as before. Doh!

To fix this, my colleague wrote a Python script (to be run outside of LW) that deletes all displacement nodal connections within a scene file, resulting in:
CellSpline_400cells_clothFXed_disconnectedNodes.lw s

-> Yay! fast loading!

but

-> bug/issue: UI is sluggish. Open Cmd History and see Layout calling "refresh" forever.. nonstop. With more normal .mdd files, this would make the UI practically unbearable to work with as I suppose Layout is constantly refreshing the load of the full .mdd file. Even now with small ones it is annoying but it can become prohibitive with realistic .mdd sizes (remember, these are low poly objects .mdd'ed for just 3 frames total)

Bah, so no true workaround either.. it does load fast due to the disconnected nodes though, yet the UI refreshing poses a new issue to battle with. Clearing/exiting/managing objects in this scene is even worse than before.

109974

edit: bugreports sent on both of the above