Can I get to the stream behind LWLoadState?

[ewinemiller]

Hi folks,

I've got a node shader where when I originally wrote the save code, I did not wrap all the parameter functions with LWSAVE_BEGIN and LWSAVE_END. This works fine for saving objects and scenes, but doesn't work for using the nodes as displacement or exporting and importing nodes because there's no name on that input.

Basically the save files look like this

{ VariantParameter
3
0
{ ParameterValue
1
0
}
}
{ VariantParameter
3
0
{ ParameterValue
1 1 1
0
}
}

instead of this

{ Noise
{ VariantParameter
3
0
{ ParameterValue
1
0
}
}
}
{ Base Color
{ VariantParameter
3
0
{ ParameterValue
0.74901962 0 0
0
}
}
}

So I'd like to fix that, but so far I haven't had any luck doing that without breaking compatibility with existing saved files. It looks like that LWLoadState object wraps a stream of some sort and if I could get it to move backwards or reset to the beginning of a block, I think i'd be good, but I don't see anything like that. Here's what I've tried, any other suggestions?

1. Use LWLOAD_ID first to see if I have a named block. Works good to identify new format, but leaves old format in a bad state because it wants LWVParmFuncs.load called first. Can't reset the stream so done.
2. Use LWLOAD_FIND to see what's the first block I have. Again, works good to identify new format, and takes me to the first block I did wrap in the old format, but then I can't go backwards to pull my parameter values.
3. Tried saving a throwaway parameter value without wrapping in the new format. On load, create a dummy parameter, evaluated it immediately, and if it's a magic number for its value, it's the new formation, else old. This actually seems to work, but the parameter functions getVal doesn't like being called from there and you end up with a stack corruption run time error.
4. Tried using LWLoadState.read to just see what was ahead of me, but again that pushes the stream ahead so everything down the line is broken.
5. Tried using LWVParmFuncs.load first, check the error code, if successful old, else new, but I seem to end up in a bad state so can't LWLOAD_FIND doesn't work for the new format.

Thanks,
Eric

 

[Jarno]

This is why it is a good idea to always version your save data. I usually have a top-level block wrapping all the save data, with a version block in it, like:

{ MyPlugin
    Version 2
    { SomeOtherBlock
        ...
    }
    SomeMoreData 42
    ....
}

LWLoadState is intentionally designed to require minimal capabilities of the stream that it controls. Particularly it does not require the stream to be random access, so there is no rewinding.

Trying to jump around the stream is probably a bad idea, as it would mess up the state machine that tracks the blocking.

Maybe for this version of the plugin you save both the old data and the new data, and on loading replace the old with the new if there is new data. Then with the next version of the plugin drop support for the old format and only save the new format. That way the users have a transition path.

---JvdL---

 

[ewinemiller]

This is why it is a good idea to always version your save data.

Hindsight and all. Would have been really good to see that kind of best practice in the SDK samples.

Trying to do a transition version is still ultimately the same problem. I don't know how to start reading my data block, without seeing my data block. I need to either do a raw LWVParmFuncs.load(old) or a LWLOAD_FIND(new) first and if I choose wrong, the stream is broken and I can't read anything more successfully or go back and start over. I think the only way is this could be helpful is that I post a conversion version that's really not meant to do anything else except load old files, then save them so you can install the version that is pure new format. The translator couldn't read anything it saved. That's incredibly awkward for my clients, but might be the only way out of it.

I did try to save two copies (basically spit out the new wrapped parameters at the end of the old format) hoping that with a wrapped name, the node connections would be picked up, but ignoring the extra param doesn't do anything and reading the same parameter a second time on load seemed to break all the connections.

Is there any kind of information (metadata, etc.) that could be persisted and examined with a param function that doesn't require a call to getVal?

Thanks.

 

[Jarno]

What I'm suggesting is that you first have all your old format data, then the new format data. When loading, first load in the old format data like with the previous version. You know how many vparms there will be, so you can load in that many blindly. Then do a find for the new data. If the new data is there, then that will succeed and you ignore what you have loaded in so far and use the new data instead. If there is no new format data after the old format data (i.e. an old version file), then the find will return failure because there is no more data in the block.

The dummy vparm idea should also work, provided it has been set up correctly. The dummy vparm may have to be created outside of loading (such as during plugin creation). It is possible that the getVal() is crashing because some setup is otherwise deferred until loading is finished.

---JvdL---

 

[ewinemiller]

So I learned a little more, got close, but not the whole way.

1. Your suggestion about just saving both formats gets me closer. The problem I was hitting when i tried it before is that the second time you read a param (e.g. read Noise in old format block, then later in new format block, read it again), the whole load gets trashed. So now I have a bunch of dummy params that I load the old format into. If I don't hit a block of new stuff, then I copy them to the real params. That gets me backwards compatibility and the ability to save and load nodes. However it fails in the nodal object deformation stuff. That one won't load the nodes without a clean new format style save. With any of the raw param saves it doesn't load. This is actually the workflow my customer is really looking for so not quite there.

2. getVal still mangles the stack even if it's created outside of load. I tried moving my dummy param there and even just called it on one of my known good params and it has the same problem. Since the raw param saving doesn't play nice with nodal object deformation, this looks like it wouldn't have helped outside of getting a cleaner save and load.

So I think I'm back to two plug-ins as a transition to the new format. First plugin just loads old and saves new. Second only talks new. There's only one person asking for a fix for deformation so I may just give the new format one to them as a one off and leave everyone else on the backwardly compatible version.

Thanks for your insight.

Regards,
Eric