PDA

View Full Version : MORE Dynamics Troubles



DrStrik9
02-18-2016, 01:54 PM
I'm pretty sure you all are getting tired of my LW dynamics difficulties, but wait! -- There's MORE ... How much would YOU pay for ...

I have been under the impression (apparently mistakenly) that once you calculate dynamics, you can make changes to the scene, as long as you don't change any of the dynamic objects.

This is apparently wrong. -- ??

I just spent almost 6 hours on a sim calc, prior to saving the scene (another 40 minutes). After this, I set up cameras and lights, and then turned dynamics back on, and it immediately begins to RE-CALCULATE dynamics, which of course, wipes out almost 7 hours of work from yesterday, in the previously-calculated and saved dynacache.

I know ... I should have exported and imported MDD, but I had so much trouble with it previously, that I decided to trust that dynacache was safe as long as no objects were changed.

What am I missing?

DrStrik9
02-18-2016, 02:02 PM
And now ...

The scene is corrupted AGAIN. Attempting to open it crashes Layout -- AGAIN. This is the FOURTH dynamics scene that has been corrupted in 2015.3 in the last two weeks.

I think I may need to give LW dynamics a good long rest ... :-(

Yet another bug report is being sent to LW3DG.

mummyman
02-18-2016, 02:35 PM
The only thing I can say to maybe help the saving part, is if I calculate dynamics, I turn them off globally "Enable Dynamics" before I save my scene, so you don't have to wait as long. But it's a crappy thing that's happening to you. Wish I knew more!

DrStrik9
02-18-2016, 02:53 PM
In attempting to send in yet another bug report with all the content, their site is apparently down for the last half-hour + ... it's stuck on "sending request to lightwave3D.com ..."

-- I think maybe LW3DG is tired of hearing from me about this horror of a bug. :devil:

DrStrik9
02-18-2016, 03:09 PM
The only thing I can say to maybe help the saving part, is if I calculate dynamics, I turn them off globally "Enable Dynamics" before I save my scene, so you don't have to wait as long. But it's a crappy thing that's happening to you. Wish I knew more!

Me too! :D The first time you save the scene after calculating dynamics, you MUST leave it on, because this is how dynacache is saved (which took 40 minutes in this scene, after 5 hours 50 minutes for sim calc). I did turn dynamics off prior to saving the scene every time after that. I only turned it back on to render the dynamics. I only wish I had backed up the scene before turning dynamics back on. That way I could have at least salvaged the lighting and camera stuff before the scene committed suicide.

Seriously, I'm going to forget all about LW dynamics until it is addressed. LW3DG has not responded to any of my bug reports on this horror as of yet ... after four scenes destroyed themselves, this does not feel like a good sign, especially when they responded immediately this morning to a small feature request.

But I'm still pretty pi$$ed off at the moment ... I will calm down eventually.

mummyman
02-18-2016, 03:32 PM
Oh man.... what kind of stuff are you working on? Dang.. can't you break it out somehow. Or does it all have to be in shot.

jboudreau
02-18-2016, 04:02 PM
I'm pretty sure you all are getting tired of my LW dynamics difficulties, but wait! -- There's MORE ... How much would YOU pay for ...

I have been under the impression (apparently mistakenly) that once you calculate dynamics, you can make changes to the scene, as long as you don't change any of the dynamic objects.

Yes the way it's suppose to work is if you move any of the dynamic objects then it will recalculate the dynamics. Also if you have any object in the dynamics list that has dynamics but are turned off if you delete them from the list or delete them from the scene it will cause dynamics to recalculate. (this I feel is a bug)

This is apparently wrong. -- ??

I just spent almost 6 hours on a sim calc, prior to saving the scene (another 40 minutes). After this, I set up cameras and lights, and then turned dynamics back on, and it immediately begins to RE-CALCULATE dynamics, which of course, wipes out almost 7 hours of work from yesterday, in the previously-calculated and saved dynacache.

I'm failing to understand something here. Setting up cameras and lights should not cause the dynamics to recalculate. Also if once you finish with all your dynamics etc, if you make a dynamics folder in you scene folder and you save your scene for example Dynamics_Project_Master.lws. This will save your scene and a dynacache file called Dynamics_Project_Master.dynacache in your dynamics folder. Once that's done save your scene again and call it Dynamics_Project_001.lws this will save another file in your dynamics folder called Dynamics_Project_001.dynacache. This way if your most recent scene crashes or gets corrupt you will always have the Master scene to fall back on saving you from having to redo the scene over and over again. So keep having a backup of your most recent scene this way you can always fall back if it gets corrupt again saving you hours of work

I know ... I should have exported and imported MDD, but I had so much trouble with it previously, that I decided to trust that dynacache was safe as long as no objects were changed.

What am I missing?

Do you have a scene to share or steps to recreate the scene? I would love to give this a try and see if I run into the same issues you are having.

Hope this helps
Thanks,
Jason

jboudreau
02-18-2016, 05:12 PM
Me too! :D The first time you save the scene after calculating dynamics, you MUST leave it on, because this is how dynacache is saved (which took 40 minutes in this scene, after 5 hours 50 minutes for sim calc). I did turn dynamics off prior to saving the scene every time after that. I only turned it back on to render the dynamics. I only wish I had backed up the scene before turning dynamics back on. That way I could have at least salvaged the lighting and camera stuff before the scene committed suicide.

Actually, I'm sorry to say but this is not correct, You do not need to have dynamics enabled for the dynacache file to be saved. Once the simulation is calculated, you can turn off the enable dynamics and save the scene the dynacache file will still be saved.

Seriously, I'm going to forget all about LW dynamics until it is addressed. LW3DG has not responded to any of my bug reports on this horror as of yet ... after four scenes destroyed themselves, this does not feel like a good sign, especially when they responded immediately this morning to a small feature request.

But I'm still pretty pi$$ed off at the moment ... I will calm down eventually.

I have a feeling that your issue is related to your lw content paths, What do you have your lightwave content path set too? Because what ever you have it set to when you save your scene is where the dynacache files save. If you are not seeing the dynacache files saved to disk with enable dynamics turned off then I think your dynacache files are being saved to where ever your lw content path is set too. This would cause the simulation to re-calculate because lightwave can not find the dynacache files.

Also another thing I found out is if you have a simulation say 100 frames (ball falling and smashing on ground) and you run the simulation and save the scene and close layout it will save a dynacache file just for those 100 frames. If you re-open the scene and say go to frame 150 the simulation will start recalculating from the beginning on frame 101. So now you will have the simulation from frame 0 - 100 (ball falling and smashing on the ground) the way you want it but on frame 101 the simulation will start over recalculating from the beginning (ball starting to fall again basically a loop).

If you do this without closing layout you can keep adding to the simulation where you left off without it starting to recalculate the simulation again.

Also have you tried locking your objects so you definitely know you are not moving them or anything, because it's very easy to move something by mistake when you thought you were moving something else

Hope this helps
Thanks,
Jason

DrStrik9
02-18-2016, 07:19 PM
Jason: I always am very careful to set content directory properly. I have saved dynamics scenes before that were previously calculated, and then with dynamics turned off and scene is saved, no dynacache is saved. I can then turn dynamics back on, resave the scene, and dynacache is saved in the Dynamics folder, within the content directory. So I'm confused by what you say about this. I have consistently had a different experience than what you describe.

To be clear, I did NOTHING to any of the dynamic objects in this scene, and LW still recalculated the sim when turning dynamics back on, even though it was calculated and saved before (sim calc took almost 6 hours).

I understand how to select objects, move them, etc., (I don't use automatic keyframes in scenes like this, but only manually-set keyframes, so unwittingly moving a dynamic object just isn't going to happen) and I did none of these things. ALL I did was add lights, cameras, and background elements. I also did not extend the scene length, having done this before a few times, prior to final sim calc and scene+dynacache being saved.

I did use your "master" scene method (with different filenames) - which took nearly an hour to save each version. I saved two, and continued work on the second one. But after the scene had to be stopped from another unexpected 6-hour sim calc, I replaced the dynacache of the working scene from backup, and at this point the scene crashes Layout on opening. So all that work finishing the scene for rendering is lost.

Thanks for thinking this through with me; I really do appreciate it, but something is not right with LW dynamics. Having four major scenes in two weeks kill themselves is a bad record for any feature.

And with no response from LW3DG on multiple bug reports, I'm moving on from large physics setups. I may do small dynamics things, but I will no longer be pushing the envelope with dynamics in Lightwave.

jboudreau
02-18-2016, 07:45 PM
Jason: I always am very careful to set content directory properly. I have saved dynamics scenes before that were previously calculated, and then with dynamics turned off and scene is saved, no dynacache is saved. I can then turn dynamics back on, resave the scene, and dynacache is saved in the Dynamics folder, within the content directory. So I'm confused by what you say about this. I have consistently had a different experience than what you describe.

something isn't right then, I just set up a scene and saved it with dynamics off and it saved the dynacache file no problem. Very strange. I wonder if all these problems is related to MAC vs PC. I use a PC.

To be clear, I did NOTHING to any of the dynamic objects in this scene, and LW still recalculated the sim when turning dynamics back on, even though it was calculated and saved before (sim calc took almost 6 hours).

I understand how to select objects, move them, etc., (I don't use automatic keyframes in scenes like this, but only manually-set keyframes, so unwittingly moving a dynamic object just isn't going to happen) and I did none of these things. ALL I did was add lights, cameras, and background elements. I also did not extend the scene length, having done this before a few times, prior to final sim calc and scene+dynacache being saved.

Again this is strange because I did a simulation and moved lights, cameras, added new objects etc nothing made the dynamic re-calculate except for what I mention in my previous post

I did use your "master" scene method (with different filenames) - which took nearly an hour to save each version. I saved two, and continued work on the second one. But after the scene had to be stopped from another unexpected 6-hour sim calc, I replaced the dynacache of the working scene from backup, and at this point the scene crashes Layout on opening. So all that work finishing the scene for rendering is lost.

No not exactly what I mean. the way you are doing it might and might not work depending on how many changes were made to the scene. What I was talking about is before doing any work save the scene as another filename and keep doing this often this way you can go back to your previous scene. I know it's tough because it's taking so long to save. Hope this makes sense

Thanks for thinking this through with me; I really do appreciate it, but something is not right with LW dynamics. Having four major scenes in two weeks kill themselves is a bad record for any feature.

I agree their are some bugs with it but most of the bugs you are referring too, I just don't see in the PC version of lightwave 2015.3. I'm after doing a large sim to see if I get the same issues you are having. Have you tried re-creating the scene but on a smaller scale so it takes way less time to calculate, save etc. This way you can see if you still have all these issues.

And with no response from LW3DG on multiple bug reports, I'm moving on from large physics setups. I may do small dynamics things, but I will no longer be pushing the envelope with dynamics in Lightwave.

I understand that can be quite frustrating


Hope this helps
Thanks,
Jason

DrStrik9
02-18-2016, 08:56 PM
Yes, after a large sim destroyed itself in 2015.3, I scaled it down and it did not have these problems. Then I rebuilt the larger scene in 11.6.3, and it worked for awhile (I was able to render half of what was planned). I thought I was home free, but it eventually destroyed itself also. This is how I ultimately discovered the png bug. So having learned to avoid png's, I built an even larger scene in 2015.3, and everything worked until I turned dynamics back on again, and this is the result. 2015.3 handles the scene beautifully, all the way through rendering. Dynamics is where the problems are. For the fourth time, the scene was destroyed, and crashes Layout on open, like all the other destroyed scenes. -- SO many wasted days, with very little to show for it.

I guess it could be a Mac issue. It does sound like there are operational differences between Mac and PC dynamics. I'm just tired of bashing my head against it.

jboudreau
02-18-2016, 09:55 PM
Yes, after a large sim destroyed itself in 2015.3, I scaled it down and it did not have these problems. Then I rebuilt the larger scene in 11.6.3, and it worked for awhile (I was able to render half of what was planned). I thought I was home free, but it eventually destroyed itself also. This is how I ultimately discovered the png bug. So having learned to avoid png's, I built an even larger scene in 2015.3, and everything worked until I turned dynamics back on again, and this is the result. 2015.3 handles the scene beautifully, all the way through rendering. Dynamics is where the problems are. For the fourth time, the scene was destroyed, and crashes Layout on open, like all the other destroyed scenes. -- SO many wasted days, with very little to show for it.

I guess it could be a Mac issue. It does sound like there are operational differences between Mac and PC dynamics. I'm just tired of bashing my head against it.

Something definitely doesn't make sense here, I just did a simulation which took about 30min to simulate it has around (140,000 polygons) . I saved the scene and it took only a couple of min to save which created a a 2GB dynacache file.

132464132465132466

I'm not sure how complex your simulation is but something must be going on with the mac version of lightwave to take 6 hours to simulate and 40min to save.

How big are your dynacache files?
How many polygons were in your simulation?

Thanks,
Jason

jeric_synergy
02-18-2016, 11:19 PM
:thumbsup: for the supportive and constructive (?) dialog in this thread.

It'd be really nice if LWG would give DrS9 some closure.... :(

DrStrik9
02-19-2016, 06:41 AM
Something definitely doesn't make sense here, I just did a simulation which took about 30min to simulate it has around (140,000 polygons) . I saved the scene and it took only a couple of min to save which created a a 2GB dynacache file.

132464132465132466

I'm not sure how complex your simulation is but something must be going on with the mac version of lightwave to take 6 hours to simulate and 40min to save.

How big are your dynacache files?
How many polygons were in your simulation?


Jason: Yes, with the size of your scene and the times for calc and save you list, there is DEFINITELY something wrong with the Mac version of Layout 2015.3, regarding dynamics.

My last destroyed scene had 389,376 polys for a total of 64,896 "boxes." The dynacache file was 2.15 GB in size. Actual calc time was 5 hours 50 minutes, and saving the scene (with dyn ON) took 40 minutes.

Since LW3DG's bug report function has been down for a day, I'm emailing everything in, which requires VERY slow uploads to Google Drive for the scene, objects, images, and dynacache.

Here are the few renders I was able to complete during this two-week nightmare: https://youtu.be/sKdq7Rno1sw

After this, I'm taking a break from LW in general.

jboudreau
02-19-2016, 10:50 AM
I don't blame you sounds like a nightmare.

After doing some testing you are absolutely right on huge scenes like yours and mine the dynacache files are not being read correctly. I took and saved my scene after all the dynamics were calculated. It created a 2GB dyancache file. Once I re-opened my scene the dynamics started re-calculating again. So basically it completely ignores the dynacache file. Why I'm not sure nothing was changed. The only thing I can think of is somehow when you save your scene the objects in your scene are being saved but something is changing. So this is causing the dynacache file to think that something has changed in the object so it re-calculates the simulation. This does not happen on smaller bullet dynamic simulations which is very strange.

What I'm going to do is save my object as a master file. Then save it again under another name. I will run the simulation and then save the scene saving the objects as well. I'm going to then compare the master file object with the scene object (which should be the same object) this way I can see what has changed. They should be exactly the same but my suspicion is they will be different. I will keep you posted on what I find out

I will keep you posted on what I find out

Thanks,
Jason

DrStrik9
02-19-2016, 02:27 PM
Thanks, Jason.

It seems before we do anything else after sim calc, must we first export and import MDD? (I didn't do this before, because MDD has its own issues, which frankly I didn't try to chase down). Or do you think this would have similar problems as we're having with dynacache, with objects mysteriously being saved, or whatever bizarre errors are happening with large sims? I guess my MDD question is: could the objects be mysteriously saved (with no changes, except save date), and MDD still work properly? If not, then as it stands, there is apparently "no way to get there from here."

The only other remote possibility I can think of would be to create the entire scene, with texturing, lighting, cameras, backgrounds, etc, then save it, and only THEN do a sim calc, and NOT RE-SAVE that scene (after dynacache is saved) until all rendering is completed. But frankly, I like to make camera changes and such, AFTER the sim is calculated, so even this workflow would not really be acceptable to me.

jboudreau
02-19-2016, 02:37 PM
Thanks, Jason.

It seems before we do anything else after sim calc, must we first export and import MDD? (I didn't do this before, because MDD has its own issues, which frankly I didn't try to chase down). Or do you think this would have similar problems as we're having with dynacache, with objects mysteriously being saved, or whatever bizarre errors are happening with large sims? I guess my MDD question is: could the objects be mysteriously saved (with no changes, except save date), and MDD still work properly? If not, then as it stands, there is apparently "no way to get there from here."

The only other remote possibility I can think of would be to create the entire scene, with texturing, lighting, cameras, backgrounds, etc, then save it, and only THEN do a sim calc, and NOT RE-SAVE that scene (after dynacache is saved) until all rendering is completed. But frankly, I like to make camera changes and such, AFTER the sim is calculated, so even this workflow would not really be acceptable to me.

Yes that's what it's looking like, After my simulation I baked the .mdd files and then saved the scene. I didn't do anything else but close the scene and re-open it where the dynamics started re-calculating again. I compared the two objects and the files match 100% so saving the scene is not changing the objects in anyway. Can you tell me what issue you were having with baking the .mdd files? I haven't had any issues with this at all. I even used the .mdd files on a different scene with the same object and it worked great.

You can do this but I don't think it's the re-saving the scene that's the problem. It's happening after only saving the scene once and re-opening it. If I do a small simulation and save the scene and re-open it then their is no recalculation. I'm suspecting something is going on once you reach a certain amount of polygons in your simulation. I did a simulation with 50,000 polygons and no issues with the dynacache files, or with the scene recalculating the simulation. I'm testing it with 100,000 polygons at the moment, I'll let you know what I find out

Thanks,
Jason

jboudreau
02-19-2016, 04:12 PM
Okay here is what I found out. I made a simulation starting with 11,000 polygons and kept increasing them as I went. Everything worked great. The dynacache files were saved and when I re-opened the scenes the scene did not recalculate the simulation. The strange thing though when I got to 200,000 polygons I saved the scene. The dynacache file saved (It was just over 2GB) but when I went to re-open the scene the simulation starting re-calculating again. It never happened with scenes that were 130,000 - 150,000 etc.

So here is what's happening if you have a simulation scene that has around 200,000 polygon or more and creates a dynacache file of 2GB or more the scene will open but the simulation will start recalculating again. It ignores the dynacache file all together. So from what I can see is the dynacache file has a limitation or something. One of the test scenes had a dynacache file at around 1.9GB and it worked. So it looks like the limitation is 2GB. Another thing that is really funny is all the dynacache files that
will not work anymore and that cause the scene to recalculate even ones that are from completely different sceness are all the same size.

4 different scenes have 4 different dynacache files with the following sizes below, The funny thing is they all say 2,097 as if their is a limitation. DrStrik9 what were the sizes of your dynacache files? I'm wondering it they were 2,097,??? kb

- 2,097,999 Kb
- 2,097,999 Kb
- 2,097,802 kb
- 2,097,190 kb

It looks like when we are saving the scenes that the dynacache file will not save more than 2GB of information even if the simulation should have a bigger dynacache file so this must be what's corrupting the scene and causing it to re-calculate.

Hope this helps
Thanks,
Jason

jboudreau
02-19-2016, 07:58 PM
Okay I definitely just verified their is a problem with the dyancache system in lightwaves bullet dynamics.

Any bullet scene no matter how many polygons will recalculate the simulation after re-opening the saved scene when the dynacache file reaches 2,097,000 kb

I made a scene with 18,000 polygons and I kept simulating the scene every 300 frames, Then I saved the scene and re-opened making it re-calculate another 300 frames etc. I did this until the dynacache file reached 2,097,000 kb. When the dynacache file reached a size of 2,092,000 kb the scene file open with no problems and was using the dyancache file no recalculation. I then opened that scene and calculate a few more frames this made the dynacache file to reach a size of 2,097,000 kb. I saved the scene and re-opened it. BANG!! The scene opened but this time it wasn't reading the dnaycache file and it started recalculating the scene again.

The scene that has a dynacache file of 2,092,000 kb works
The same scene that has a dynacache file of 2,097,000 doesn't work causes the scene to recalculate the simulation

This happens in both 11.6.3 and 2015.3. I think from what I revealed here is that possibly the LW3DG has put a limitation on how large the Dynacache files can reach. 2,097,000 or larger

Hope this helps
Thanks,
Jason

DrStrik9
02-19-2016, 08:01 PM
Wow, excellent work, Jason! :)

If it's OK with you, I will copy your text and send it as an addendum to my bug report emailed this morning (since the bug report thingy is currently not working).

We might actually be getting somewhere! :D

jboudreau
02-19-2016, 08:05 PM
Wow, excellent work, Jason! :)

If it's OK with you, I will copy your text and send it as an addendum to my bug report emailed this morning (since the bug report thingy is currently not working).

We might actually be getting somewhere! :D

Thanks man, Absolutely no problem, I will put in a bug report as well. I'm curious what are the sizes of your dynacache files? I have a felling they are that same magic number of 2,097,000 or higher

Also just to add to this even after re-calculating another 100 frames the dynacache file will not get any bigger than 2,097,000 kb

Thanks,
Jason

DrStrik9
02-19-2016, 08:23 PM
Thanks man, I will put in a bug report as well. I'm curious what are the sizes of your dynacache files? I have a felling they are that same magic number of 2,097,000 or higher

Yup. 2.15 GB.

I also wonder if the scene-destroying thing that's been plaguing me is a separate problem, not necessarily directly associated with the dynacache file size limit ... so I need to be prepared to have additional issues, even after this particular bug is squished.

I went ahead and quoted you, adding your text to my earlier bug report. I think LW3DG needs all the help they can get ... :thumbsup:

--

I just got an email back:

"Issue LWB-2034 has been marked as CLOSED.
There is indeed a 2GB or so limit on the dynacache file size. It's because it is a 32bit IFF format.
This is fixed in LW2016 where the dynacache format is 64bit."

Holy cow.

jboudreau
02-19-2016, 08:31 PM
Yup. 2.15 GB.

I also wonder if the scene-destroying thing that's been plaguing me is a separate problem, not necessarily directly associated with the dynacache file size limit ... so I need to be prepared to have additional issues, even after this particular bug is squished.

I went ahead and quoted you, adding your text to my earlier bug report. I think LW3DG needs all the help they can get ... :thumbsup:

--

I just got an email back:

"Issue LWB-2034 has been marked as CLOSED.
There is indeed a 2GB or so limit on the dynacache file size. It's because it is a 32bit IFF format.
This is fixed in LW2016 where the dynacache format is 64bit."

Holy cow.

Are you kidding me, Why and the F$$% couldn't the LW3DG provide this in their manual instead of us having to spend days and days and you weeks of trying to figure out what the problem is. GRRRRRRRRRRRR

So then basically what Lino was telling us about the scene shouldn't re-calculate and being told we could do complex simulations was not exactly true. Their is no way you could do any sort of complex simulation if the dynacache file is limited to 2GB unless you bake .mdd files before saving the scene, and depending on how complex the simulation this could potentially crash your scene and you loose everything which I think is what you ran into. Unbelievable!!

Thank you so much for letting me know this. Glad I could help get to the bottom of this issue.

In all my tests I never had the scene corrupt where it couldn't be opened again. Is this the problem you were having?

Thanks,
Jason

jwiede
02-19-2016, 08:56 PM
I just got an email back:

"Issue LWB-2034 has been marked as CLOSED.
There is indeed a 2GB or so limit on the dynacache file size. It's because it is a 32bit IFF format.
This is fixed in LW2016 where the dynacache format is 64bit."


That is an easily-foreseeably flawed design. Such an implementation suggests both questionable design methodology, and grossly inadequate testing.

I believe LW3DG owes recent LW customers a better answer than "Buy more Ovaltine!", and ASAP.

jboudreau
02-19-2016, 09:08 PM
That is an easily-foreseeably flawed design. Such an implementation suggests both questionable design methodology, and grossly inadequate testing.

I believe LW3DG owes recent LW customers a better answer than "Buy more Ovaltine!", and ASAP.

Tell me about it. This is ridiculous that this was not mentioned. I'm pissed and poor DrStrik9 must be furious over this especially with the issues he's been having over the past two weeks because of this issue. All because the LW3DG failed to mention this very important information regarding dynacache files.

Also how can they say that lightwave is 64-bit when it sounds like it's a mix between 32-bit and 64-bit

Thanks,
Jason

jwiede
02-19-2016, 09:23 PM
Tell me about it. This is ridiculous that this was not mentioned. I'm pissed and poor DrStrik9 must be furious over this especially with the issues he's been having over the past two weeks because of this issue. All because the LW3DG failed to mention this very important information regarding dynacache files.

Also how can they say that lightwave is 64-bit when it sounds like it's a mix between 32-bit and 64-bit

More to the point, NOBODY should be using any IFF-like 32-bit-based format/protocol for storing modern 3D datasets with geometry-scaled sampling, PERIOD. Absolutely 100% foreseeable issue, particularly in a 64-bit-architecture. The only thing more shameful than the dynacache format choice being accepted through design review, is that the resultant issue itself wasn't caught during developer module testing, or any subsequent testing.

jboudreau
02-19-2016, 09:33 PM
More to the point, NOBODY should be using any IFF-like 32-bit-based format/protocol for storing modern 3D datasets with geometry-scaled sampling, PERIOD. Absolutely 100% foreseeable issue, particularly in a 64-bit-architecture. The only thing more shameful than the dynacache format choice being accepted through design review, is that the resultant issue itself wasn't caught during developer module testing, or any subsequent testing.

I know you are right, I did some research on the iff format and it turns out this format was created way back in 1985 back in the Commodore and Amiga days.

Interchange File Format (IFF), is a generic container file format originally introduced by the Electronic Arts company in 1985 (in cooperation with Commodore/Amiga) in order to facilitate transfer of data between software produced by different companies. IFF files do not have any standard extension.

Thanks,
Jason

jboudreau
02-19-2016, 10:25 PM
I'm pretty sure you all are getting tired of my LW dynamics difficulties, but wait! -- There's MORE ... How much would YOU pay for ...

I have been under the impression (apparently mistakenly) that once you calculate dynamics, you can make changes to the scene, as long as you don't change any of the dynamic objects.

This is apparently wrong. -- ??

I just spent almost 6 hours on a sim calc, prior to saving the scene (another 40 minutes). After this, I set up cameras and lights, and then turned dynamics back on, and it immediately begins to RE-CALCULATE dynamics, which of course, wipes out almost 7 hours of work from yesterday, in the previously-calculated and saved dynacache.

I know ... I should have exported and imported MDD, but I had so much trouble with it previously, that I decided to trust that dynacache was safe as long as no objects were changed.

What am I missing?

Hi DrStrik9

I just wanted to let you know something in case you are not aware. If you ever open your scene that has simulation, for example one that would have a smaller then 2GB dynacache file (since we know what happens there) and the simulation starts re-calculating because you moved an object by mistake or click on a keyframe that didn't have calculations yet, As long as you don't save the scene it should not affect your dynacache file. All you need to do is just close the scene without saving and re-open the scene and everything should be back to normal your previous simulation should still be there without having to re-calculate again.

Hope this helps
Thanks,
Jason

js33
02-20-2016, 12:10 AM
I was going to chime in that 2gb file sounded like a 32-bit limitation but you guys beat me to it already. IFF was the major file format on the Amiga so either LW3DG didn't see a problem leaving some elements as 32 bit when LW went "64 bit" or they didn't have a 64 bit replacement ready.

spherical
02-20-2016, 03:31 AM
I was going to chime in that 2gb file sounded like a 32-bit limitation but you guys beat me to it already. IFF was the major file format on the Amiga so either LW3DG didn't see a problem leaving some elements as 32 bit when LW went "64 bit" or they didn't have a 64 bit replacement ready.

Yep. I was recalling a 2GB limit that applied somewhere in the past. Wasn't able to nail it down before the response came back about the .iff type used. That said, archiving a periodic series of file copies would have saved a lot of this anguish. All eggs in one basket and such. Just sayin'....

jboudreau
02-20-2016, 04:26 AM
Yep. I was recalling a 2GB limit that applied somewhere in the past. Wasn't able to nail it down before the response came back about the .iff type used. That said, archiving a periodic series of file copies would have saved a lot of this anguish. All eggs in one basket and such. Just sayin'....

Yes that's exactly what I did to get to the bottom of this problem. The reason it took so long is the simulations take a long time to get the dynacache file to reach 2GB. A scene with only 18,000 polygons took 2500 frames of simulation to achieve the 2GB limit and I had to be careful that I didn't simulate too far a head because once it would hit that limitation of 2GB and you had no idea when this would be until you saved the scene it would corrupt your scene file. Luckily using the archive method I could go back to the previous scene that I had just calculated. I also had to figure out if it was a polygon count issue or if it was a dynacache issue so High polygon count simulations of 200,000 / 300,000 polygons were taking a very long time to calculate.

In DrStrik9's situation it didn't matter if he archieved his scene or not because his simulations were somewhat complex (6 - 7 hours to calculate) So every time he calculated the simulation it always would exceed the 2GB limitation. So even if he saved multiple versions he was just saving corrupted version every time. This is why he thought that moving lights, and cameras and setting up textures and backdrops was causing the dynamics to re-calculate. Not realizing it wasn't that at all but that he had went over the 2GB size limitation on his dynacache files.

I'm sorry to say but it wasn't archiving that would of saved DrStrik9 a lot of this anguish. If the LW3DG would of let us know by putting this valuable information in their manual, then that alone would of saved him from going through all of this.

Thanks,
Jason

DrStrik9
02-20-2016, 09:20 AM
... the resultant issue itself wasn't caught during developer module testing, or any subsequent testing.

It seems we (customers who paid for the privilege) have been doing that testing now. :screwy:

jeric_synergy
02-20-2016, 09:34 AM
Are you kidding me, Why and the F$$% couldn't the LW3DG provide this in their manual instead of us having to spend days and days and you weeks of trying to figure out what the problem is. GRRRRRRRRRRRR
Jason
And THIS is why we need a DYNAMIC, USER-UPDATEABLE, CROWD-SOURCED MANUAL, so that such information doesn't GET LOST.

Right now someone else is probably doing the exact same task, and without looking at this forum, how would they ever know?

jboudreau
02-20-2016, 09:37 AM
And THIS is why we need a DYNAMIC, USER-UPDATEABLE, CROWD-SOURCED MANUAL, so that such information doesn't GET LOST.

Right now someone else is probably doing the exact same task, and without looking at this forum, how would they ever know?

Is there anyway we could possibly do this ourselves, because you know if we have to wait for the LW3DG to do anything about it unfortunately we might be waiting for a very long time

Thanks,
Jason

jboudreau
02-20-2016, 10:33 AM
Did you guys know you could edit the actual lightwave manual that comes with 2015.3? All you do is open the PDF manual in adobe acrobat and choose edit / edit text and images. I wonder if this is how we could possibly keep the manual up to date with information that users have figured out that is missing from the manual.

Here is a quick example of me adding the dynacache information to the bullet dynamics dynacache page in the manual

132504

Jeric what exactly did you have in mind, how would it work?

Thanks,
Jason

DrStrik9
02-20-2016, 10:51 AM
And THIS is why we need a DYNAMIC, USER-UPDATEABLE, CROWD-SOURCED MANUAL, so that such information doesn't GET LOST.

Right now someone else is probably doing the exact same task, and without looking at this forum, how would they ever know?

Believe me, share your frustration and I see your point, but ... the result is something like Wikipedia ... which isn't all that bad, but probably not THE definitive source of all wisdom and knowledge. I think Ben Vost does a VERY admirable job putting the LW docs together. The latest 2015.3 pdf is by FAR the best LW documentation I have ever seen personally. I have used the 2015.3 pdf to my great benefit hundreds of times. The problem with any resource document is that it can never really be complete (your point again). In the same way that there is "always one more bug," there is always one more thing that "could have been included" in the manual. My concern about a group-maintained LW reference is that it can easily become a diversified mess, and by its very nature, it must be managed by someone (or a group) who is well-funded and continually honing down all knowledge related to the subject, which changes constantly. I, by myself, can't really fully digest everything that is discussed just in this forum, let alone process all of it and organize it into a salient structure.

Let's face it. Developing something like Lightwave3D is a multi-faceted effort, being done by far too few brilliant developers who, although they are deeply committed to the process, and who are inspired to work harder than the average person, can only do so much. They are divided on all fronts: chasing bugs, adapting others' work into their workflow, developing and integrating what will sell the next rev, etc., etc. The opposite of how LW3DG is doing this is a monstrosity like Alias Wavefront (Alias System Corporation), or whatever they're called these days: a huge bureaucracy, which costs the user huge dollars to participate in using the product. And it has as many issues as LW does, if not more!

In retrospect, the only reason we're having this discussion is because LW3DG had the wisdom to implement and maintain these forums. I was driven here by my multi-week horrific experience with trying to create large bullet sims. (This is my curse/calling in life: to push the envelope until the envelope breaks, and thereby assist in defining (along with people like jboudreau) the broken boundary. -- It's a sort of "idiot-calling," if you will, but I believe a necessary role in development.)

So I don't have the answer, if it exists, to "incomplete documentation," but I think we're on the right track, in being the LW community that we are. In some ways, we're the "janitors" who pick up all the loose ends, point them out, and report our difficulties. I see this as an essential part of the group-development process. We're the ones who help the next guy NOT have to experience the nightmares that we experience. And this is going to continue ... even if we don't want it to.

I get the perspective that "they should have done a better job," (that's why they call it "development.")... but never forget: it makes no sense to beat ourselves up (or someone else) over a problem we only discovered yesterday. We can start beating ourselves (or others) up if the problem is ignored and left unfixed.

</soapbox> :D

I just also hope that when they make bullet 64-bit, that it is also multi-threaded. :hat:

jboudreau
02-20-2016, 11:05 AM
Believe me, share your frustration and I see your point, but ... the result is something like Wikipedia ... which isn't all that bad, but probably not THE definitive source of all wisdom and knowledge. I think Ben Vost does a VERY admirable job putting the LW docs together. The latest 2015.3 pdf is by FAR the best LW documentation I have ever seen personally. I have used the 2015.3 pdf to my great benefit hundreds of times. The problem with any resource document is that it can never really be complete (your point again). In the same way that there is "always one more bug," there is always one more thing that "could have been included" in the manual.

I agree with everything you said, The reason I was so frustrated is yes this was found yesterday by myself but it's not really a bug. This was known information that the LW3DG knew but failed to mention to us or have it provided in the manual. I understand their are bugs and we will always find something new, but if they are aware of things and limitations not bugs they should be providing us with this information. This would of caused you a lot less grief, time and stress had you known about this limitation.

Thanks,
Jason

jeric_synergy
02-20-2016, 12:07 PM
Ben (BeeVee) does indeed to a very admirable job of documenting an enormous program. But I feel that his Sisyphean efforts (and believe me, I mean that) should be the sturdy foundation on which the enhanced and dynamic documentation would be based.

IOW, we START with BeeVee's efforts, and ADD to them. PLUS, and I've said this before but perhaps not in this thread, any additions would be moderated by vetted volunteers from the LW community. So it's not just a free-for-all, it's good solid information. The kind of information you could have used, and that Jason kindly fully characterized.

And I'm in agreement with you that fully volunteer doc efforts tend to be lacking-- for which example I'll simply point to the Blender dox, which while they may be complete-ish, are a complete mess. (I'll stop, because I've typed this all many times.)

jwiede
02-20-2016, 02:55 PM
I was going to chime in that 2gb file sounded like a 32-bit limitation but you guys beat me to it already. IFF was the major file format on the Amiga so either LW3DG didn't see a problem leaving some elements as 32 bit when LW went "64 bit" or they didn't have a 64 bit replacement ready.

We're talking about a format that is part of their Bullet integration, both design and implementation made quite recently (and long after the 64-bit shift), not some ancient legacy feature code.

While listing the limitation in the docs would have avoided this (somewhat), such a limitation should NEVER have made it into a modern-implemented feature in the first place. This is much more about poor decision-making during design, and failure to adequately test that the functionality was suited for complex, real-world production environments, than about a documentation issue (IMO).

These cache files scale based on the source geometry, and as demonstrated, even a moderate-scale object (200K polys) can easily result in a >2GB cache. Even if the limitation had been documented, all that would really mean is that the feature was documented to just not be all that useful. Accepting a 2GB limit in such a format is the real bug here, particularly in a "new-ish" feature implementation with "geometry x time"-like sampling scaling.

jwiede
02-20-2016, 03:31 PM
BTW, even if we accept the 2GB limitation as a design flaw (versus a "bug" proper), there absolutely are serious bugs involved:

Layout generates "illegal" cache files, without error. Accepting the design, Layout shouldn't be attempting to write >2GB files at all, but is doing so. Generating a file that violates the file type's protocol/format is a bug (and a nasty one), the code should instead give an error.


Layout appears to be corrupt itself when attempting to read the illegally-formatted files generated in #1, instead of giving an error. If Layout isn't catching creation of such files , then the code dang well should at least be able to recognize an illegally-formatted cache file, and cleanly fail with an error. Crashing when reading ill-formatted input files is a bug (also nasty), Layout should cleanly terminate the read and generate an error instead.

No matter how you look at it, there are still serious bugs depicted -- bugs that even basic, minimal corner-case testing should have caught.

(edit)

I'm glad to hear they addressed the 2GB limit for the next version, but I'd feel a lot better if LW3DG's response had also mentioned addressing the bugs above. Just lifting the limitation isn't really adequate by itself, they also need to beef up the error- and sanity-checking in the read/write code (as high priority as fixing the limitation). It's about having a defensive mindset when coding, something still too scarce in LW.

jeric_synergy
02-20-2016, 11:44 PM
Crashing when reading ill-formatted input files is a bug (also nasty), Layout should cleanly terminate the read and generate an error instead.[/list]
It's about having a defensive mindset when coding, something still too scarce in LW.
Indeed. There shouldn't BE such a thing as my "poison file" label-- incoming data should be parsed in such a way that it cannot crash the app.

This standard may not be possible with scripts, but any data file should be parsable.

DrStrik9
02-22-2016, 12:17 PM
It's about having a defensive mindset when coding, something still too scarce in LW.

Not being a coder, but being a "totalist" kind of person in my own work, I concur. If development attitude is "just do enough to get by for today," then we're probably in for a long series of problems like this.


And I'm in agreement with you that fully volunteer doc efforts tend to be lacking-- for which example I'll simply point to the Blender dox, which while they may be complete-ish, are a complete mess. (I'll stop, because I've typed this all many times.)

I also find the Blender docs to be almost unusable. (I’d give them a C- or maybe D+, while Ben’s work on LW docs would get an A.)

But … since this dynacache disaster with bonus scene corruption, I’ve been driven (and I mean DRIVEN) to seriously look at Blender for dynamics, and therefore by necessity to Blender in general. Yes, its interface is weird and for me, virtually impossible to navigate without following endless tutorials, but the results with physics, once you get there, are stunning. — And quite different than Lightwave’s, which seem a bit "simplistic" by comparison.

The interesting thing about Blender’s physics is that it calculates in very near REAL TIME. After 5:50:00 sim calcs in LW that killed the dynacache and the scene as well, this is quite attractive to me, making it worthwhile to wade through all Blender's uber-weird interface issues, and slowly implanting muscle-memory for navigating scenes — a rather painful process, to be honest.

… But not nearly as painful as the previous two weeks bashing my head against LW dynamics.

So … (as traitor-like is this may seem to some) I’m spending more time trying to grok Blender than I’m spending in LW this last week or so …

… and … more shall be revealed …

jwiede
02-22-2016, 12:23 PM
So now LW2015 will just be left forever in a state where dynamics is seriously broken for anything beyond fairly simple scenes (due to the corrupt dynacache and endless recalc)?

That's pretty shady, IMO, given "improved dynamics" was one of the main LW2015 selling points in the first place.

These dynamics bugs merit a (free) bug fix update release for LW2015.

DrStrik9
02-22-2016, 01:26 PM
So now LW2015 will just be left forever in a state where dynamics is seriously broken for anything beyond fairly simple scenes (due to the corrupt dynacache and endless recalc)?

That's pretty shady, IMO, given "improved dynamics" was one of the main LW2015 selling points in the first place.

These dynamics bugs merit a (free) bug fix update release for LW2015.

Well, not exactly, meaning not "forever." They are finally making dynamics 64-bit, according to bug-report-related emails from LW3DG, in LW 2016. So this "should" allow us to calculate dynacache beyond the 32-bit limit of ~2K (which right now, with large sims, overwrites the dynacache when it gets beyond ~2K, which corrupts both the dynacache file and the scene file as well).

Because of the file format (IFF) being 32-bit, they don't consider this a bug (!), and so have closed the bug report, saying it will be fixed in 2016, by becoming 64-bit. I guess my question is: why fix it if it isn't broken? (It's broken, so it's a BUG.) Or lazy programming, a development oversight, whatever.

I do see this as a killer-sized bug in 2015.3, since (1) it destroys the dynacache and scene files, and (2) LW is ostensibly "64-bit." -- This glaringly reveals how a "64-bit" application can unfortunately have segments that remain 32-bit.

How many 32-bit segments of Layout will remain, even in 2016? Who knows. :-(

jwiede
02-23-2016, 12:16 PM
Well, not exactly, meaning not "forever." They are finally making dynamics 64-bit, according to bug-report-related emails from LW3DG, in LW 2016. So this "should" allow us to calculate dynacache beyond the 32-bit limit of ~2K (which right now, with large sims, overwrites the dynacache when it gets beyond ~2K, which corrupts both the dynacache file and the scene file as well).

But again, that requires customers to pay and upgrade to the next version (LW 2016) in order to receive the fix, it does not fix LW _2015_. Meanwhile, LW2015 customers who do not upgrade to 2016 will apparently be stuck (forever) with seriously broken dynamics, no 100% reliable way for them to know a priori whether their save can be reloaded, nor any viable alternative approach.

Snosrap
02-23-2016, 12:38 PM
At some point it just makes more sense for the devs to move on. I'm afraid that LW2015 will receive 0 more service packs. I'm okay with that as long as new architecture allows for faster development, fixes and feature advances.

Kaptive
02-23-2016, 01:36 PM
Though I kind of agree with your points John, how many serious users of Lightwave will remain on 2015? I might be totally wrong, but I'd imagine that anyone who isn't doing 3d as a core activity would be unlikely to use bullet to its fullest degree. No guarentees with that of course, but the amount of people affected would probably be small. Not that it makes the problem you highlight go away, but from a business point of view (LW3DGs) the amount of resources required to fix it for a small amount of people would be out of balance. When you say forever... it's the idea that in 5... 10... 15...?? years time, people will still be using 2015.3... but if they stopped there, then perhaps they are looking elsewhere or their artistic direction is changing. I'm far from a rich man, but the upgrade price continues to be very affordable imho, so if you still have an interest in producing 3d stuff, then the upgrade price isn't a massive one to save up for.

Of course, if enough people... who feel that they don't want to upgrade, but feel very likely to push bullet to this breaking point make enough noise, then maybe it'll happen. Anyways, I'm just playing devils advocate.

Like I say, I'm not disagreeing, just saying that the cost versus users affected might make it a very unlikely fix... and I'd actually understand why it wouldn't be at this point (with the new version coming out).

jwiede
02-23-2016, 01:58 PM
Though I kind of agree with your points John, how many serious users of Lightwave will remain on 2015?

Two important things to remember about LW 2016:


New render engine, no legacy surfacing/rendering support (or related plugins), need to use older version to render legacy scenes.


New infrastructure engines (renderer & UGE, perhaps others), thus potentially significant infrastructure (read as: app-wide) stability issues.

I see ample basis for belief that LW 2015 will continue to be used by customers for quite some time, and that many customers may be slow at migrating to LW 2016 for day-to-day usage.

Kaptive
02-23-2016, 02:26 PM
Two important things to remember about LW 2016:


New render engine, no legacy surfacing/rendering support (or related plugins), need to use older version to render legacy scenes.


New infrastructure engines (renderer & UGE, perhaps others), thus potentially significant infrastructure (read as: app-wide) stability issues.

I see ample basis for belief that LW 2015 will continue to be used by customers for quite some time, and that many customers may be slow at migrating to LW 2016 for day-to-day usage.

Fair points. It'll be interesting to see how that pans out.

DrStrik9
02-23-2016, 04:21 PM
Those are fair points, John. I still use 11.6.3 once in awhile, when I experience issues with 2015.3. I might wait awhile to upgrade to 2016, until most major issues are resolved, as with 2015 -- unless there's a beta program of some kind, which I would be happy to participate in ... after my dynamics nightmare over the past few weeks, I'm a bit shy about paying for another beta app.

Having said that, I'm still exploring Blender for physics, and the learning curve is an odd, long, and winding road ... but it is clear that Blender' physics are all 64-bit, and multi-threaded, which is a HUGE performance boost over LW. Plus, the physics really do seem "more realistic," which may simply be a better set of defaults, or possibly a newer Bullet implementation, I don't know yet. Of course, in order to benefit from this, one must deal with Blender's interface. :D And then there's Blender's renderer ... oh my. :oye:

It's always something ...

Verlon
07-12-2017, 12:36 PM
Yup. 2.15 GB.

I also wonder if the scene-destroying thing that's been plaguing me is a separate problem, not necessarily directly associated with the dynacache file size limit ... so I need to be prepared to have additional issues, even after this particular bug is squished.

I went ahead and quoted you, adding your text to my earlier bug report. I think LW3DG needs all the help they can get ... :thumbsup:

--

I just got an email back:

"Issue LWB-2034 has been marked as CLOSED.
There is indeed a 2GB or so limit on the dynacache file size. It's because it is a 32bit IFF format.
This is fixed in LW2016 where the dynacache format is 64bit."

Holy cow.

My Lightwave is 2015.3
Where can I download this LW2016 you speak of?
More to the point, objects *I* modeled, *I* textured, and *I* rendered in prior versions of Lightwave (64 bit) are not loading now. The current object is barely 1 meg in size. There is no simulation of any kind. It is just an object (and textures from photoshop). IT throws this same error.

DrStrik9
07-12-2017, 05:00 PM
My Lightwave is 2015.3
Where can I download this LW2016 you speak of?
More to the point, objects *I* modeled, *I* textured, and *I* rendered in prior versions of Lightwave (64 bit) are not loading now. The current object is barely 1 meg in size. There is no simulation of any kind. It is just an object (and textures from photoshop). IT throws this same error.

This is an old thread. My last post was from early 2016. But until the next LW update, the thread remains somewhat relevant. I'm pretty sure "Lightwave 2016" became "Lightwave NEXT," which is still "in development," with very little news from NewTek on its progress and/or release for quite awhile now. I'm just waiting, fooling around with Blender, and hoping this isn't the end of Lightwave. Only time will tell.

As far as I know, the LW object format has not changed for many years. ALL my old LW objects still load, from LW 5.6 through 2015.3. I suspect you may have some kind of data or disc error on those objects that won't load. Hopefully you have a decent backup prior to what might be corruption. Good luck!

jeric_synergy
07-14-2017, 10:36 AM
"Dynacache", not LWO. And LWG itself confirmed the limitation.

jwiede
07-14-2017, 07:14 PM
(deleted, just realized it was a necrothread)

antoniomitalia
05-29-2019, 03:32 PM
It is not about the 2GB limit. Got the same problem with a 130kb file. Just export the model in OBJ format from Modeler and Layout will open it, no size limits.
If your model is NURBS, just subdivide 2 times (faceted + metaform) before exporting.