PDA

View Full Version : Why the focus on Collada file format is bad.



Vincenzo
04-19-2009, 08:02 PM
1. Collada is a fat format and will over double the amount of disk space to store models.

2. Collada is to flexible!!! Thats right! Collada stores files in a self descriptive format that allows the storage of almost any type of model. The problem is that each program takes its own approach to rendering and this can be reflected in the format it likes to store its object info in, so just because other apps can read LW Collada format object files, that doesnt ensure LW collada object file will translate precisely to other apps objects.

For example, LW lwo objects can contain VMAD sections which associate a different UV value for each polygon that contains a point for a given VMAP, which allows the possiblity of associating multiple UV values to a single point within a single UV Map. This allows LW to handle texture seams well, but the problem is that DirectX and Obj files take a different approach to handling texture seams. DirectX and associates a single UV value to each Vertex for each uv map (texture coordinate) and it handles texture seams by allowing a wrapping interpolation renderstate (note this is not same as the wrapping address mode which is a sampler state), and assumes the texture seams occur at the 0.0, 1.0 boundaries in UV space. Thus a LW artist is not required to create the texture seams at the 0, 1boundaries of UV space. LW will handle this fine, but when the artist wants to import the LW into say a game engine or Maya, there are problems. This is a case where FLEXIBILITY IS BAD. Note if the artist creates the texture seams at the 0 and 1 borders in UV space the texture coordinates can be translated into DirectX well.

Now LW VMADs can be represented in Collada, and other apps that can read Collada files can read LW VMADs, but it doesnt mean that VMADs translate well into the other apps objects. This is a case where COLLADA offers the HYPE of INTEROPERABILITY, but doesnt deliver.

I personally think that interactive graphics are becoming a bigger and bigger part of the graphics market. If LW wants to be an important part of this market then it needs improve its support of OBJ files which map well to DirectX and OpenGL rasterization. Perhaps provide a checkbox during object creation, if this object will be used only in LW (then allow full flexibilty) or if it should be expected to be imported into other 3D apps (then reduce the flexibilty so the object can be imported precisely by other apps).

Thats all, Vincent Burnett

lwanmtr
04-19-2009, 08:34 PM
I am one of the few who will probably agree with you. I would much rather see an improvement in the obj and lwo formats as well as see a flexible collada export..
I've mentioned file size in other posts, but most folks seem happy with bloated file sizes.
I have never had a good collada transfer from LW to another app...I use FBX if I need to go to Maya or Motion Builder. In fact, I have never even been able to import Collada into Maya2009 at all.

Collada is the buzzword of the day, though, so yaay.

Sekhar
04-19-2009, 08:38 PM
Does the standard allow for zipped/compressed versions? I vaguely remember reading something about it... I'd played a bit with Papervision 3D (the Flash thing) using COLLADA objects and used the built-in ZIP API to zip/unzip because the files were prohibitively large for online use otherwise.

AdamAvenali
04-19-2009, 08:39 PM
I have never had a good collada transfer from LW to another app...I use FBX if I need to go to Maya or Motion Builder.
i have never tried to go from lightwave to maya, but have tried to go from maya to lightwave, which proved unsuccessful. the model would come into layout perfectly fine, but the camera and lights were totally jacked.

i dont think file sizes are as concerning as before for most people with 1 Tb drives becoming the norm, though i try to stay as small as possible because uploading to rendercore can take forever, sometimes longer than the actual render haha

IMI
04-19-2009, 08:40 PM
I'll chime in on the side of "let's have better OBJ support", for what it's worth.
I haven't learned much about Collada yet, and it may very well be the thing for the future. As toothless old men we may sit around on rocking chairs sipping lemonade on the porch and laughing at the days when OBJ was still important, before Collada took over, but we're not there yet. ;)

lwanmtr
04-19-2009, 08:46 PM
On the nore of file sizes...I made a collada with an object (8mb lwo) after the scene went to collada it was nearly 50mb...thats a huge difference to me and says that the format should be rethought about for at least the internal LW format.

IMI
04-19-2009, 08:47 PM
i dont think file sizes are as concerning as before for most people with 1 Tb drives becoming the norm, though i try to stay as small as possible because uploading to rendercore can take forever, sometimes longer than the actual render haha

Doesn't the file size also affect the RAM usage?

lwanmtr
04-19-2009, 08:50 PM
yeah..though I'm not sure how much

IMI
04-19-2009, 09:01 PM
Well OBJ files can get pretty large too, since they're text files.

I have to wonder though if Collada isn't just a pipe dream. Aren't all the 3D app makers just going to implement it as they see fit anyway? I'm not sure that Autodesk is particularly concerned about if a LW Collada file works right in Maya or not, for example. And will another 3D app maker change/rewrite his whole Collada import/export to coincide with the changing whims of another 3D app?

Vincenzo
04-19-2009, 09:48 PM
IMI;

Thats a good point. Collada is an xml based descriptive file format. LW can export files as Collada files, but other apps dont have to read LW Collada files.

My point was that different apps often take different approaches to solving issues (LW VMADS are an example) and as a result can assign information to objects that other apps do not associate with their objects. Thus even though different apps can express their objects via Collada, the objects of different apps still dont translate well to each other. Thus Collada support doesnt mean that the import/export of objects is perfect.

IMO if Lightwave was the dominate 3D app, then it could do what it wanted and other apps would have to adapt, but LW is not the dominate app and it needs to make certain that assets created in it are usable in a FX studio or a game studios pipeline. Collada/xml is just a word that sounds good to managers and does not ensure interusability of assets. Its just a bloated format. Also the flexibility that it provides can even be a hindrince to asset exchange from different apps.

PS. I do believe there is a binary OBJ version.

lwanmtr
04-19-2009, 10:01 PM
Yeah, obj's can get pretty large too..but it's not a big hit because I usually delete the obj version once im done with it...lwo has always been a very efficient format, which is one of the reasons I have questioned the switch to Collada since they announced it.

I'm hoping that they will keep a legacy format for those who dont need collada.

beverins
04-21-2009, 08:44 AM
After Newtek have Core v1.0 out the door, - or perhaps sometime in Q4 when the program is gaining strength - I think ONE avenue to this would be that Newtek could make a flexible DAE saver.

While nothing will magically compensate for different formats, I think that certain things CAN be done in terms of having underlying algorithms that would be able to translate to various flavors of DAE.. both in and out.

Basically... Radio buttons on the side of an Export DAE operator. Save DAE as Direct X Compatible... Blender Compatible.. Maya Compatible..

To be sure maybe this isn't even possible, but if Newtek are going to make DAE the default they need to make DAE as robust as humanly possible, with as many options as can be - perhaps "hidden" behind "advanced options"

Titus
04-21-2009, 12:09 PM
Doesn't the file size also affect the RAM usage?

Why? it's only a description of your scene (it has the same points, faces and textures like the usual LWO), it doesn't matter the size when you have a clear exchange format, if the size is a problem NT can implement a binary and zipped collada format. RenderMan RIB was thought as the ubicuous 3D format, but Pixar decided to not let everyone to participate on their developement.

Titus
04-21-2009, 12:13 PM
Well OBJ files can get pretty large too, since they're text files.

You can encapsulate an OBJ just like Blender does with their fluid simulation files. Even they are compresed, the simulation playback is very fast.

DiscoBurgess
04-22-2009, 08:47 AM
Like a bitmap, meshes should be the same size in memory whatever their file format. JPEGs aren't compressed in memory, so they take up more RAM than their file size, for example.

COLLADA, LWO and the rest all describe vertexs, surfaces and polygons, and those things take up the same number of bytes however you load them.

Titus
04-22-2009, 09:50 AM
Like a bitmap, meshes should be the same size in memory whatever their file format. JPEGs aren't compressed in memory, so they take up more RAM than their file size, for example.

Some file formats store more information than is needed. For example RenderMan requires tiff with mipmaps, this means the original image plus smaller versions. But yes, in general a jpg or tif file with the same resolution takes the same memory once they are on the program's buffer.

Nemoid
04-25-2009, 08:03 AM
not an expert of collada, but if i understand correctly the collada implementation is an internal problem of how Lw manages various items like vertex maps, objects and so on.

But, may also be that Lw CORE will adopt a different way to deal with this, differently from what Lw currently does, in that way being more standard, and similar to other "high end" apps ? Does this make sense?

Its obvious that Autodesk will take care about interoperability between its own apps, and so their implementation of collada will lead others within the market.

The same can be valid for .obj format which is their format of choice at least for now...

Vincenzo
04-26-2009, 08:05 PM
What I was referring to was that just providing Collada support does not ensure combatability between apps, or that an object in one program will map exactly to an object in another program. So Collada doesnt ensure that maya will be able to import lightwave objects for example. Collada is also a fat format. So focusing on Collada just makes objects take up more disk space but does not ensure interoperability between apps.

To provide an example of a feature that LW provides an object, but other apps do not, I consider LW VMADs. LW VMADs (discontinous vertex maps )are an example of a feature that LW can associate with an object, but objects in other apps dont have. VMADs allow one to associate different UV values to a single point for each polygon that contains that point. They are good for dealing with seams in textures. Other apps dont have this feature. Thus when importing a LW object (.lwo LWO2) into game engines or other apps this can cause problems. If the LW artist was smart and knew that the LW object was going to be used in a game engine or other app he or she can create the texture seams to occur along the 0.0, and 1.0 boundaries in UV space. The UV map should then be able to be imported into other apps easily, but if the artist did not know about this then they may create the UV seams at different places in LW which can cause problems when importing the LW object into other apps.

Nemoid
04-27-2009, 02:40 AM
Thanks for the explaination !:)
I am sure that Nt will find a way to deal with this: focusing only on Collada format or whatever, is not so clever.
Multiple options are always the best. A well made .obj export is a good solution either, so i'd simply work on that either.

lwanmtr
04-27-2009, 05:23 AM
I am still hoping they keep .lwo and .lws, at least as an option.

IMI
04-27-2009, 05:39 AM
I am still hoping they keep .lwo and .lws, at least as an option.

Yeah, I'd imagine they'll have to, at least for a while. Can you imagine how pissed off we'd all be if we couldn't open our scenes and objects even from last year, in CORE? ;)

But I guess the worst we could expect eventually is that their Collada just won't work in any other apps, but will work fine in Core. Not that that would be a good thing, but it wouldn't be much different than what we have now.

lwanmtr
04-27-2009, 03:30 PM
The thing with Collada is that it is the newest multi-app transfer method. And while it promises to be a very good system, it is not implemented in every app and even the apps that do support it do it differently...Poser, for instance, supports Collada, but even going between LW and Poser using it is a pain.

If you want to use the other apps as a reference...They still use their own formats natively, but have Collada, Obj, Fbx, etc. as export options.

My feeling is that even if Core uses Collada natively, there should be a well written lwo/lws import and export system...Lets face it, alot of use will continue to use those because they are very efficient formats, and has been noted in this thread have some features that even the super high end packages dont...

The UV map thing...I often keep my UV's away from the absolute edge of the uv space..and while I dont generally tile uv's it's nice that LW will automatically adjust for that space when I do.

I do also think that Collada will no doubt be replaced with yet another interchange format in the near future and that, yup, we'll be right where we are now..

FBX, Obj are still the best options for going between most packages, Collada is ok, but there is still no real standard for it (and Core wont have enough influence to be able to set that standard).

IMI
04-27-2009, 03:45 PM
I do also think that Collada will no doubt be replaced with yet another interchange format in the near future and that, yup, we'll be right where we are now..

FBX, Obj are still the best options for going between most packages, Collada is ok, but there is still no real standard for it (and Core wont have enough influence to be able to set that standard).


Yeah, I thought it seemed more than a little strange when I read that CORE would be using Collada as its file format. I thought at first maybe I just read it wrong, or maybe they said it wrong, that it would be its primary interchange format, not just its de facto 3D format.

Because then, and now, that just seems like a strange decision. I think we were all expected to jump up and down and cheer or something. ;)

But after seeing what they did to OBJ this past year...

lwanmtr
04-27-2009, 03:52 PM
Yeah, it really is a strange decision..but they wanted to have unified file format to go along with the unified app...But really, they need to stick with lwo/lws at least as an option..because, I'd bet good money that the Collada format is gonna come back to bite 'em in the butt later on....plus, why lose the ability to transfer files to 9.6? Sure 9.6 cant do alot of what Core will, but I do alot of work for clients who probably wont go Core for a long time (since 9.6 does what they need it to do).

Lightwolf
04-27-2009, 04:11 PM
I actually think it is a good move.

If file size is an issue then a gziped variant would be a decent addition.

And of course, Collada does not ensure cross platform compatibility... not even if an app only writes what is defined in the spec. It depends very much on both sides of the equation, the writing and the reading app (then again, this goes for any file format).

But... Collada provides a well defined structure based on one of the most often used language to define structures. Which means that there's tons of parser and tool out there to extract something from the lump of data.
And while it doesn't ensure compatibility with other apps... it makes it easier to add - and I don't see anything wrong with making things easier.

Heck, you can use just about any XML based tool chain to extract something from a Collada file... try that with LWO/LWS (which, to make matters worse, are two different beasts to parse).

Cheers,
Mike

IMI
04-27-2009, 04:20 PM
There's no question Collada is more flexible and customizable, but I think all we're talking about is what appears to be dropping support for the .lwo format.

But I guess it's safe to ASSume that won't happen until Core can stand entirely on its own. As I understand it, LW will still be needed in conjunction with CORE, at least for a while.

However it's still not clear exactly which "LightWave 3D" will be CORE's companion app, and I suppose it could also be assumed that it might be intentionally engineered to *not* work with "Legacy" LW 9.6, so people *have* to buy CORE to get the "LightWave 3D" that IS compatible...

Lightwolf
04-27-2009, 04:25 PM
There's no question Collada is more flexible and customizable, but I think all we're talking about is what appears to be dropping support for the .lwo format.
What I doubt you'll see is LWO being extended to support the feature of CORE (i.e. construction history).
That would certainly be a waste of time, especially as LWO would need to be completely revamped anyhow to work around current limitations (just think of the max. storage size allowed to surface node graphs, which some people have hit already).


But I guess it's safe to ASSume that won't happen until Core can stand entirely on its own. As I understand it, LW will still be needed in conjunction with CORE, at least for a while.
I'd expect native Collada files in CORE and excellent LWO support as a bridge. But I would expect some loss of information when saving a CORE native model as a LWO.

Cheers,
Mike

lwanmtr
04-27-2009, 04:26 PM
Exactly, not saying that they shouldnt pursue Collada, just that keeping the lwo system intact is a good idea..If they can get a stable Collada going thats fine.

On the filesize thing..using compressed files slows things down..especially if you need to access assets from different files.

lwanmtr
04-27-2009, 04:28 PM
What I doubt you'll see is LWO being extended to support the feature of CORE (i.e. construction history).
That would certainly be a waste of time, especially as LWO would need to be completely revamped anyhow to work around current limitations (just think of the max. storage size allowed to surface node graphs, which some people have hit already).

I'd expect native Collada files in CORE and excellent LWO support as a bridge. But I would expect some loss of information when saving a CORE native model as a LWO.


Yup, thats to be expected...lwo cant support some of the things, unless they rewrite the lwo..but it's not necessary....people will use lwo to save space and/or to use in older LW, where some of the Core features dont exist.

Lightwolf
04-27-2009, 04:29 PM
On the filesize thing..using compressed files slows things down..especially if you need to access assets from different files.
Considering that current CPUs can decompress faster than most drives can read... and that most of the slow down doesn't come from reading the file but processing them and converting them to internal data structures... I'd doubt it.
Otherwise just about any LW scene should load in 5-10s... and I've seen a couple that take 5-10 minutes to load, with fairly small files (a total of 20 MB).

Cheers,
Mike

IMI
04-27-2009, 05:17 PM
I'd expect native Collada files in CORE and excellent LWO support as a bridge. But I would expect some loss of information when saving a CORE native model as a LWO.

Cheers,
Mike


Well yeah, of course. In the same way that Modo can open a .lwo object - it comes into Modo with most of the surface data, and all of the vertex maps - selections sets, UVs, morphs and so on... intact, but obviously can't do nodes and procedurals, for example. Although luminosity, textures, and all the basic stuff is preserved. And even a subpatch .lwo is pretty much about the same.

And the same goes when exporting from Modo as .lwo, just in reverse. So I'd expect CORE to do about the same with .lwo.

IMI
04-27-2009, 05:21 PM
However it's still not clear exactly which "LightWave 3D" will be CORE's companion app, and I suppose it could also be assumed that it might be intentionally engineered to *not* work with "Legacy" LW 9.6, so people *have* to buy CORE to get the "LightWave 3D" that IS compatible...

You [email protected], get out of conspiracy mode!

Pay attention - if you buy CORE you will HAVE that new "Lightwave 3D" to go along with it.
Idiot. :o

Lightwolf
04-27-2009, 05:22 PM
So I'd expect CORE to do about the same with .lwo.
I honestly hope it will do more. Otherwise it will be just about as useful to me as modo is... i.e., marginally and only used when there is no alternative way in Modeler (but that goes with how I tend to work... very, very iteratively, refining on all parts of a project until the deadline hits me).

Cheers,
Mike

Silkrooster
04-27-2009, 06:28 PM
I haven't had a chance to look at the collada specs, but i would assume it works similarly to the obj file format. In that the main portion of the format is standard that all apps must obey that portion of the format. Then at the end it has a location to store custom perimeters that each app can call their own. And It would be up to that company to publicize those perimeters to guarantee they do not clash with other apps.

Now as for calling the format fat, I hope you are calling it that because it can expand. Normally when I hear the word fat and it has to do with the drive I think of the fat table. As we all know the format can not dictate how it is stored on the drive, that is the responsibility of the drives OS.

Lightwolf
04-28-2009, 05:02 AM
Now as for calling the format fat, I hope you are calling it that because it can expand.
Well, it is fat because it uses a lot of space to store data that could be stored in a slimmer way.
XML by itself tends to make files larger, it's a tradeoff between flexibility and storage efficiency.

Cheers,
Mike

Silkrooster
04-29-2009, 12:08 AM
Ok I see what you mean. If they added some kind of compression that should alleviate that some.

Nemoid
04-29-2009, 05:48 AM
in the tech faq they clearly say collada is default format, explaining that

"In classic LightWave, evaluation orders are effectively fixed; this makes importing COLLADA and FBX files from other applications difficult at times, especially if rotation orders have been rearranged. The same would hold true for deformation evaluation order.

LightWave CORE™ has complete flexibility in evaluation order of any system. This will better facilitate data interchange with other applications in the pipeline. Rotation order, deformation order, etc., are completely under user control, offering the most versatility for problem solving."

to me this means they'll try to make things in a way that interoperability will be more easy than what one would obtain with COLLADA compiled for current Lw structure.

also if collada is well projected , maybe compiling a sort of traslator for importing the whole data structure into other apps could be easier than making a lwo - lws importer or similar plugin.

there are also. obj and .fbx in the game, so that options will be alot.

As for LW=, i think they'll make in a way we will be able to exchange files between it and CORE. BTW, since LW 9.6 has not history or modifier stack those datas will be wiped out on exprt, and model freezed for sure. but, may be other datas like textures and vertex maps will be preserved.

when current Lw will be discontinued, then the Lwo format will be very probably be supported for awhile, then progressively abandoned.

Silkrooster
04-30-2009, 12:24 AM
The only way i could see NT abandoning the lwo format is if a script was created to batch convert files from lwo to collada. I imagine the same would hold true for lws.

lwanmtr
04-30-2009, 12:33 AM
im sure that would work flawlessly...lol

collada
05-01-2009, 04:34 PM
1. Collada is a fat format and will over double the amount of disk space to store models.

COLLADA documents are comparable in size to other text encoded formats like Maya ASCII. The compressed form of COLLADA, .ZAE archives, are comparable in size, and often smaller, then binary formats.



2. Collada is to flexible!!! Thats right! Collada stores files in a self descriptive format that allows the storage of almost any type of model. The problem is that each program takes its own approach to rendering and this can be reflected in the format it likes to store its object info in, so just because other apps can read LW Collada format object files, that doesnt ensure LW collada object file will translate precisely to other apps objects.

It's true that COLLADA is very flexible and as a standard intermediate language for 3D digital content that is an appropriate design consideration. A language is most helpful when it can fully express what the author/speaker (end-user) desires. Extensibility and the ability to add new vocabulary (features) is part of that too. Adopting standard languages (communication) is an important way of reducing the costs associated with authoring, cummunicating, etc..

At collada.org, we encourage vendors to document their COLLADA extensions (https://collada.org/mediawiki/index.php/Portal:Extensions_directory) so that they can be shared and implemented by others. As yet, Newtek has not documented any extensions.

To help improve conformance and interoperability, Khronos is working on releasing the COLLADA Conformance Test Suite (CTS) this summer. Look for announcements around SIGGRAPH time.

ps: Great discussion

Cheers,

lwanmtr
05-01-2009, 04:44 PM
COLLADA documents are comparable in size to other text encoded formats like Maya ASCII. The compressed form of COLLADA, .ZAE archives, are comparable in size, and often smaller, then binary formats.

I use Maya a bit and find that it's file formats are too large too IMO..hehe.

Vincenzo
05-05-2009, 03:24 AM
Collada;

I was just pointing out that having a flexible format doesnt always thinks easier to import in some cases. It would be interesting to see file size comparisions between Obj files and Collada files for objects with differ face and vertex counts. Also I am certain Collada will be better at surfaces than obj files, but there are still gonna be problems importing surfaces between different apps.

Also autodesk isnt interested in compatibility imo. Theywould rather have developers write export plugins for their apps than make a format that is easily imported into other apps.

Red_Oddity
05-05-2009, 05:43 AM
Seriously, why would you want trade off a format that can be easily parsed so it will fit without too much effort in most pipelines to a binary format that needs specific loaders that need to be compiled for every format update just to spare a couple of megabytes when the average 1TB disk today costs less than 100 euros a piece?

Anyone who has done some pipeline work or file parsers knows the advantages an ascii format holds over a proprietary binary format.