Taml Custom Serialization Changes
by Melv May · 02/25/2013 (12:36 pm) · 10 comments
This is a heads-up to let everyone know that I'll soon be checking in a change into the development branch that I've been working on over the weekend and today.
A little back-story is required here first though. Very early last year when I created Taml I had a limited amount of time to complete it. I knew I needed custom serialization and so the solution I generated for that part of Taml suited the immediate requirements back then for 3SS. I did red-flag it then that the custom serialization needed to be much more sophisticated to allow for objects to persist almost anything they required but at the time, what was there worked for the objects using it.
Prior to release of T2D I wanted to make the modifications to Taml but events conspired against me (you know, only 24 hours in a day etc) and the changes I wanted didn't get it.
Recently I added the ability to have scene-controllers and I was very concerned about serializing them in a way that wasn't ugly as hell and as it turned out, I was not wrong. On Saturday I jumped on Skype with Mich and basically went over this situation and made my case for changing it. He agreed that if it was going to change, it should be ASAP.
So back to today. I have made all the relevant changes to Taml and have spent several hours testing stuff. I wanted to highlight something here though; I've not changed the whole format of Taml or anything drastic, just how the custom serialization works. I believe it's a better format for the more complex custom serialization and hopefully more natural. Whilst this doesn't sound important, whilst there are no editors, being able to understand and write Taml is more important than ever.
So to give you a better sense of what's going on, consider how a polygon collision shape is persisted currently:
The new set-up essentially allows an object to pump-out an arbitrarily large tree of nodes containing not only elements and attributes but also element text and it can now even write out SimObjects!
The above example translates into:
Another place you can see this is in certain assets but in this case I've ensure that they are backwards compatible. Here's the two different ways keys can be store in a particle:
These changes affect the binary serialization as well. I've bumped-up the version of the binary writer to v2 to indicate that.
---
For most folks, this won't make any difference however, if you've written any engine types to use the new API then you'll probably be very happy with the changes you have to make but you will have to make some minor changes. The API is less verbose as well.
I'll be merging my changes into the development branch this evening so you can all check it out.
As always, feel free to ask questions.
A little back-story is required here first though. Very early last year when I created Taml I had a limited amount of time to complete it. I knew I needed custom serialization and so the solution I generated for that part of Taml suited the immediate requirements back then for 3SS. I did red-flag it then that the custom serialization needed to be much more sophisticated to allow for objects to persist almost anything they required but at the time, what was there worked for the objects using it.
Prior to release of T2D I wanted to make the modifications to Taml but events conspired against me (you know, only 24 hours in a day etc) and the changes I wanted didn't get it.
Recently I added the ability to have scene-controllers and I was very concerned about serializing them in a way that wasn't ugly as hell and as it turned out, I was not wrong. On Saturday I jumped on Skype with Mich and basically went over this situation and made my case for changing it. He agreed that if it was going to change, it should be ASAP.
So back to today. I have made all the relevant changes to Taml and have spent several hours testing stuff. I wanted to highlight something here though; I've not changed the whole format of Taml or anything drastic, just how the custom serialization works. I believe it's a better format for the more complex custom serialization and hopefully more natural. Whilst this doesn't sound important, whilst there are no editors, being able to understand and write Taml is more important than ever.
So to give you a better sense of what's going on, consider how a polygon collision shape is persisted currently:
...
<Sprite.CollisionShapes>
<Polygon
Point0="0 0"
Point1="3 0"
Point2="3 6"
Point3="0 6">
</Sprite.CollisionShapes>
...The reason why those points are persisted like that is related to limitations in the custom serialization. Essentially it has a fixed structure for persisting a single level of elements with fields.The new set-up essentially allows an object to pump-out an arbitrarily large tree of nodes containing not only elements and attributes but also element text and it can now even write out SimObjects!
The above example translates into:
...
<Sprite.CollisionShapes>
<Polygon>
<Point>0 0</Point>
<Point>3 0</Point>
<Point>3 6</Point>
<Point>0 6</Point>
</Polygon>
</Sprite.CollisionShapes>
...A full example of all the collision shapes would be:...
<Sprite.CollisionShapes>
<Circle
Radius="2" />
<Polygon>
<Point>2 -0.45</Point>
<Point>1.9 0.2</Point>
<Point>0.5 0.65</Point>
<Point>-0.5 0.6</Point>
<Point>-2 -0.3</Point>
<Point>-2 -0.65</Point>
<Point>0.5 -0.75</Point>
</Polygon>
<Chain>
<Point>-1 0</Point>
<Point>1 1</Point>
<Point>2 0</Point>
</Chain>
<Edge>
<Point>-1 0</Point>
<Point>1 0</Point>
</Edge>
</Sprite.CollisionShapes>
...Not only are points using their own elements but they are using element text rather than attributes.Another place you can see this is in certain assets but in this case I've ensure that they are backwards compatible. Here's the two different ways keys can be store in a particle:
<FixedForce
Keys="0 0" />
<FixedForceVariation
Keys="0 0" />
<SizeXLife>
<Key Time="0" Value="0"/>
<Key Time="0.2" Value="1"/>
<Key Time="0.5" Value="1"/>
<Key Time="0.9" Value="1"/>
<Key Time="1" Value="0"/>
</SizeXLife>The custom serialization API has been streamlined to support objects writing out engine types to take away the worry of how types are formatted. This means you can write vector2, floats, bool, int, colorF, colorI (etc) and not worry about the formatting. This can be shown by writing out a color and seeing its named equivalent like "RoyalBlue" appear.These changes affect the binary serialization as well. I've bumped-up the version of the binary writer to v2 to indicate that.
---
For most folks, this won't make any difference however, if you've written any engine types to use the new API then you'll probably be very happy with the changes you have to make but you will have to make some minor changes. The API is less verbose as well.
I'll be merging my changes into the development branch this evening so you can all check it out.
As always, feel free to ask questions.
About the author
#3
I've wondered, though, why you have "Sprite.CollisionShapes" instead of just "CollisionShapes". No biggie either way, of course. I assumed it was related to having a 1-deep limitation and some trickery around that.
02/26/2013 (10:58 am)
Sweet!I've wondered, though, why you have "Sprite.CollisionShapes" instead of just "CollisionShapes". No biggie either way, of course. I assumed it was related to having a 1-deep limitation and some trickery around that.
#4
An object can have as many of these custom collections as it wishes. For instance the Scene type has a "Joints" and a "Controllers" collection so if you were to just use a child element for everything then it becomes hard to distinguish what should be an actual child added and what is for custom serialization i.e. everything becomes ambiguous. Also with this schema, Taml doesn't need to understand the types. If there were only children then it wouldn't be able to collate custom serialization elements and distinguish them from type-elements which is vital as custom serialization elements can mean anything and often have nothing to do with actual types.
This is roughly similar to how XAML works where a property with a period represents an object that cannot be represented as a string using an attribute. Inside Taml this is known as an extended-property where currently, extended properties are only used by the custom serializer.
The nice thing about it is that this way, the custom serializer can collate these custom trees and present them using a simple accessor method that is independent of the format type and it can do this after the object has been created, properties set, children added etc.
Hope that makes sense!
02/27/2013 (1:56 am)
Any child element without the period is a child object which is added to the type using the interface class TamlChildren so types like Scene and SimSet implement this so that SceneObject and SimObject types can be added respectively.An object can have as many of these custom collections as it wishes. For instance the Scene type has a "Joints" and a "Controllers" collection so if you were to just use a child element for everything then it becomes hard to distinguish what should be an actual child added and what is for custom serialization i.e. everything becomes ambiguous. Also with this schema, Taml doesn't need to understand the types. If there were only children then it wouldn't be able to collate custom serialization elements and distinguish them from type-elements which is vital as custom serialization elements can mean anything and often have nothing to do with actual types.
This is roughly similar to how XAML works where a property with a period represents an object that cannot be represented as a string using an attribute. Inside Taml this is known as an extended-property where currently, extended properties are only used by the custom serializer.
The nice thing about it is that this way, the custom serializer can collate these custom trees and present them using a simple accessor method that is independent of the format type and it can do this after the object has been created, properties set, children added etc.
Hope that makes sense!
#5
I've wanted to do this for a long time, just never had the time.
02/27/2013 (2:00 am)
As a separate comment I'd like to say that it's entirely possible to ask the engine to export a schema file the represents structure for each type so that if you use an XML editor that understands schemas then you'd get options as to what is valid and what isn't.I've wanted to do this for a long time, just never had the time.
#6
I did want to mention that in several of the other XML based 2D point representations out there (like cocos2d's and iOS's) it is common to use the format <Point>{0, 2}</Point> as well as multiple point structures like <Rect>{{0, 2}, {2, 4}}</Rect>. I have personally found those easier to parse visually when I am looking at a long list of them in an XML editor (even though it is a bit more verbose).
I just thought I would mention that in case you were interested in having it better match what seems to be the "standard".
02/28/2013 (12:29 pm)
Very cool! TAML is one of my favorite persistence libraries and I would love to see it used more widely =)I did want to mention that in several of the other XML based 2D point representations out there (like cocos2d's and iOS's) it is common to use the format <Point>{0, 2}</Point> as well as multiple point structures like <Rect>{{0, 2}, {2, 4}}</Rect>. I have personally found those easier to parse visually when I am looking at a long list of them in an XML editor (even though it is a bit more verbose).
I just thought I would mention that in case you were interested in having it better match what seems to be the "standard".
#7
The actual format comes from the console type itself and not Taml so for these types for instance which are Vector2 then you can change the console-type setter/getter to produce/consume that format.
You can see that here. Note that this has changed in the dev branch to use the supporting function inside Vector2 itself here.
The point is that you can change the format on the type itself therefore it's trivial to work with multiple incoming formats for any type so pretty simple to get that JSON-style formatting in for Vector2.
The obvious advantage using this method is that Taml doesn't care about that stuff, you can change it centrally and it'll also allow the same formatting to be used on field assignments.
03/01/2013 (12:17 am)
Thanks Matt! :)The actual format comes from the console type itself and not Taml so for these types for instance which are Vector2 then you can change the console-type setter/getter to produce/consume that format.
You can see that here. Note that this has changed in the dev branch to use the supporting function inside Vector2 itself here.
The point is that you can change the format on the type itself therefore it's trivial to work with multiple incoming formats for any type so pretty simple to get that JSON-style formatting in for Vector2.
The obvious advantage using this method is that Taml doesn't care about that stuff, you can change it centrally and it'll also allow the same formatting to be used on field assignments.
#8
if engine can support multi-asset in one taml file, i think will be more flexible. :)
03/18/2013 (11:50 pm)
a taml file just only contain an asset, such as ImageAsset, AnimationAsset, my assets folder will filled with a lot of tamls.if engine can support multi-asset in one taml file, i think will be more flexible. :)
#9
If you want multiple asset descriptions in a single file I believe you should be able to add the assets to a SimSet and then use TamlWrite(%yourSimSet, "./YourFileName.taml"); to persist the whole set. Not sure how it would like reading that back in (I'm not sure if TamlRead("./YourFileName.taml"); would actually register the contained asset descriptions with the AssetDatabase), but if not you can write a script function to re-register them on load.
The reasoning behind leaving them one description file per asset was that it makes it much easier for your customers to update their game - if you change something they only need the exact file that changed instead of a monolithic collection of asset descriptions.
03/25/2013 (8:11 am)
@Kevin - a TAML file can contain many different types of things, not just asset descriptions. At the moment that is the heaviest use of them, however.If you want multiple asset descriptions in a single file I believe you should be able to add the assets to a SimSet and then use TamlWrite(%yourSimSet, "./YourFileName.taml"); to persist the whole set. Not sure how it would like reading that back in (I'm not sure if TamlRead("./YourFileName.taml"); would actually register the contained asset descriptions with the AssetDatabase), but if not you can write a script function to re-register them on load.
The reasoning behind leaving them one description file per asset was that it makes it much easier for your customers to update their game - if you change something they only need the exact file that changed instead of a monolithic collection of asset descriptions.
#10
Just wanted to pop in and make sure this is well understood. As for the idea of multiple asset definitions in a single file, I am personally opposed to it. My own opinion is that kind of work environment has worse productivity and usability of separating asset definitions into multiple files. This will become especially true as we start developing editors. At that point, your editing is centralized through a GUI, so what does it matter if they are all in separate files? Not to mention, this is better primed for optimized project packaging and deployment.
03/25/2013 (8:20 am)
I want to make sure everyone really latches on proper terminology. [b]TAML is a persistence system[/]b, not a format. When you use TAML to write out an ImageAsset, it is currently writing out to an XML format. You can change the format to write out to binary instead. We could add a new reader/writer that could handle JSON format. Just wanted to pop in and make sure this is well understood. As for the idea of multiple asset definitions in a single file, I am personally opposed to it. My own opinion is that kind of work environment has worse productivity and usability of separating asset definitions into multiple files. This will become especially true as we start developing editors. At that point, your editing is centralized through a GUI, so what does it matter if they are all in separate files? Not to mention, this is better primed for optimized project packaging and deployment.
Torque Owner David Robert Pemberton
www.deadlyassets.com
... <Sprite.CollisionShapes> <Polygon> <Point>0 0</Point> <Point>3 0</Point> <Point>3 6</Point> <Point>0 6"</Point> <Polygon> </Sprite.CollisionShapes> ...you forgot to close the Polygon </Polygon>P.S. I much prefer this way, Its easily readable :-)