PDA

View Full Version : LW Bullet Dynamics Dynacache Limitation (Casuse scenes to re-calculate)



jboudreau
02-19-2016, 08:16 PM
Okay I definitely just verified their is a problem with the dyancache system in lightwaves bullet dynamics.

Any bullet scene no matter how many polygons will recalculate the simulation after re-opening the saved scene when the dynacache file reaches 2,097,000 kb

I made a scene with 18,000 polygons and I kept simulating the scene every 300 frames, Then I saved the scene and re-opened making it re-calculate another 300 frames etc. I did this until the dynacache file reached 2,097,000 kb. When the dynacache file reached a size of 2,092,000 kb the scene file open with no problems and was using the dyancache file no recalculation. I then opened that scene and calculate a few more frames this made the dynacache file to reach a size of 2,097,000 kb. I saved the scene and re-opened it. BANG!! The scene opened but this time it wasn't reading the dnaycache file and it started recalculating the scene again.

The scene that has a dynacache file of 2,092,000 kb works
The same scene that has a dynacache file of 2,097,000 doesn't work causes the scene to recalculate the simulation

This happens in both 11.6.3 and 2015.3. I think from what I revealed here is that possibly the LW3DG has put a limitation on how large the Dynacache files can reach which is 2,097,000 or larger

Also even after simulating more frames in the scene 100's more frames, the dynacache file will not exceed the 2,097,000 kb size (it will change the last 3 digits but never the first 4 digits (2,097). Even scenes that have more calcuations the last 3 digits are smaller so it just keeps changing the last 3 digits from 000 - 999 but will never make it go to say 2,098,000

it will give number like this

- 2,097,999 Kb
- 2,097,999 Kb
- 2,097,802 kb
- 2,097,190 kb

You can follow this thread if you want anymore information http://forums.newtek.com/showthread.php?149844-MORE-Dynamics-Troubles

Hope this helps
Thanks,
Jason

jboudreau
02-19-2016, 08:43 PM
WOW!!! Just found out that what I spent days testing and trying to get to the bottom of a problem that a user on the forum was having regarding LW Bullet Dynamics that the LW3DG knew all along but failed to mention it.

It turns out I am absolutely right regarding the 2GB file size limitation. It turns out their is in fact a 2GB limitation because the dynacache file is a 32bit iff file. Supposedly this will not be a problem in 2016 since it will be 64bit iff file.

Really LW3DG seriously!! why couldn't this be mentioned in the LW Manual. It would of taken 2 seconds to write it in there instead you have caused me and another user days and him weeks only to have corrupted scenes because of this issue.

For anyone that is building any long, high polygon count (200,000 or higher) bullet dynamic simulations as soon as it's done calculating bake your simulation to .mdd files because if your dynacache gets to that magic number of 2,097,000 your scene could be corrupted or it will cause your scene after re-opening it to re-calculate the simulation which could cause you hours and hours of re-calculating the simulation again.

Hope this helps
Thanks,
Jason

vector
02-20-2016, 12:00 AM
Good to know. Thanks for your effort.

jboudreau
02-20-2016, 05:37 AM
Good to know. Thanks for your effort.

No problem glad I could help

Thanks,
Jason

zapper1998
02-20-2016, 06:31 AM
Thank you
for the heads up.

:)

lertola2
02-20-2016, 07:49 AM
Thanks for the info. Your procedure of calculating in steps was great because it revealed what the problem was. I think I would have just done the calculation in one go and then be left with no clue why the calculation failed.

ianr
02-20-2016, 09:06 AM
I may suggest that while this is great research from yourselves & dr9strike

I may be prudent to visit http://www.bulletphysics.org/Bullet/phpBB3/

& ask Erwin Cousmans (Bullet Dev) about these short-comings?

Honestly here a feel that this is where the shortfall lies, maybe NOT with

undocumented LW3DG tests, but having said that large 'soak'(stress) tests

should be the order of the day with brought-in libraries.

What we get in LW2016 Next Bullet application on this site's releases hopefully.

jboudreau
02-20-2016, 09:19 AM
I may suggest that while this is great research from yourselves & dr9strike

I may be prudent to visit

Thanks for the link and suggestion. I'm not sure if this is a shortcoming of bullet itself, but It was stated from the LW3DG that they were using a 32bit Iff for the dynacache file format. Right there because it's 32bit it has a 2GB limitation. They have now implemented a 64-bit version in Lighwave Next (2016) I remember back in the day with a 32bit operating system you couldn't create videos any larger than 4gb. 64-bit operating systems fixed this issue. Sounds like the same type of issue

Thanks,
Jason

DrStrik9
02-20-2016, 11:02 AM
And as I pointed out in another thread, my sincere hope is that when Bullet becomes 64-bit, that it will also be MULTI-THREADED. No wonder large sim calcs take many hours (almost 6 in my case). Shouldn't this be a something over 1-2 hours on a 6-core system?

I drool when I think of the time saved.

JamesCurtis
02-21-2016, 03:39 PM
I had pointed this 2GB limit out about 3 years ago to the teck's when I was doing a sim of breaking material panels and kept having reload problems. Really was pulling my hair out!! They assured me back then that it would be addressed.

jboudreau
02-21-2016, 04:27 PM
I had pointed this 2GB limit out about 3 years ago to the teck's when I was doing a sim of breaking material panels and kept having reload problems. Really was pulling my hair out!! They assured me back then that it would be addressed.

Oh great because they are assuring us that this will be fixed in Lightwave NEXT (2016) because the dynacache file system will be 64-bit iff instead of 32-bit. I guess time will tell

Thanks,
Jason

DrStrik9
02-22-2016, 11:52 AM
I asked LW3DG directly (in bug report follow-up emails) if the 32-bit sim recalc with larger sims and scene corruption are linked. (I believe they probably are, assuming the scene file is appropriately altered at some point to address the re-calculated dynacache, which of course, is rewriting itself after ~2K, and "overflowing" -- making the dynacache itself corrupt.) I didn't get a straight answer, but I did get a "maybe/probably" kind of answer on whether Dynamics will be multi-threaded in 2016.

"I have heard that more is multi-threaded but I don't have specifics.

I'm being told that it's [dynamics is] calculating much faster - so this could be related."

-- Not much help there, really.

It becomes painfully clear that just because LW is "64-bit" that a lot of parts are still 32-bit, and only some of this old stuff is being addressed with each rev. :-(

jwiede
02-23-2016, 01:14 PM
It becomes painfully clear that just because LW is "64-bit" that a lot of parts are still 32-bit, and only some of this old stuff is being addressed with each rev. :-(

That's not really an accurate characterization of the problem, the code in question IS 64-bit code (as in, adheres to the X64/amd64 ISA) not 32-bit code (x86 ISA). It's incorrect to state that the code is "32-bit" in the sense you're using. Understand, just because code is written for X64 doesn't necessarily mean it uses 64-bit-sized types for every single value -- doing so would be inefficient, in cases where only 32-bit values are possible. The developer is responsible for understanding when 64-bit types are needed for scalability reasons, and using them, but not doing so still doesn't result in "32-bit code" per se. The term "64-bit" vs "32-bit" when referring to apps primarily refers to the target ABI of the compiler toolchain, and in that sense, the 64-bit version of LW is "wholly 64-bit code" (as in, X64/amd64 ABI code).

erikals
02-23-2016, 02:02 PM
It turns out there is in fact a 2GB limitation because the dynacache file is a 32bit iff file
even i knew this. but i guess even a coder can have a bad day...