Torque 2D MIT (Part 2)

This is part 2 of my technical blog, welcome back!

Part 1 of this blog can be found here.

Modules

Another aspect of Torque 2D that I found very frustrating when I first started using Torque was the sheer number of scripts I had to understand before I even started. There was no “hello world” scripts available but far, far worse was that the infrastructure scripts that dealt with things was confusing, coupled all over the place, had no version control, could not be loaded discreetly, blah blah blah. It was a confusing mess.

One of the driving forces of 3-Step Studio was the ability to update the product from the web. Add to this that this also had to happen for assets as well (I’ll come to those later).

I went to my trusty pen and paper and came up with a list of things that I felt needed to be addressed to allow for code modularity of TorqueScript without modifying TorqueScript itself (originally I was considering doing that). I knew that “packages” were a construct from the early days of Torque but they just didn’t cut-it for what was needed here.

Here’s part of the shopping list of features I came up with:

• Physical code isolation in separate folders i.e. a “module”.
• Module identification and versioning (Module-Id, Version-Id, Build-Id, Description)
• Module dependency controls
• Module discovery, loading, unloading and enumeration
• Module meta-data (description, type, deprecation
• Module logical grouping
• Module live-updating

I implemented all the features above and lots more. There’s a huge wiki reference document on the GGs servers that goes into a lot of detail about this system for both end-users and developers and I won’t even attempt write-up that level of detail here but I will go over the cool stuff and hopefully demonstrate how modules are not scary and are in actual fact really easy to use.

A module is just a folder. More specifically, it’s a folder with a module definition file. A module definition file is just a torque type of “ModuleDefinition” persisted using, yes you guessed it, Taml.

In XML it looks like this:

<ModuleDefinition
   ModuleId="TruckToy"
   VersionId="1"
   Description="A monster truck toy."
   Group=”Cool”
   Dependencies=”CoreModule=1,GraphicsModule=2”
   Type="toy"
   ScriptFile="main.cs"
   ScopeSet="TruckToySet"
   CreateFunction="createTruckToy"
   DestroyFunction="destroyTruckToy"
   />

The only mandatory fields are the “ModuleId” and “VersionId” (actually this defaults to “1” so I lied). The module-Id is a unique identification for the module. The version is obviously the version but you can have multiple co-resident modules of different versions. A new version of the same module indicates a newer module but also that it has breaking changes from the old one. You can load a module explicitly by its module-Id and/or its version-Id if you wish. The same goes for unloading it. When you load a module then any specified “createFunction” in executed and likewise the “DestroyFunction” is called when it is unloaded. The “ScopeSet” is just a SimSet that is automatically generated when a module is loaded and is given the name you specify here. You can use this as a mechanism to delete objects automatically when the module is unloaded because the SimSet is deleted when the module is unloaded along with any objects added to it.

Rather than load modules explicitly which requires your code being coupled to lots of module-Ids similar to the nasty “exec(“./blah.cs”)” lines you’d see littering the scripts where if you want to load something then you’d add a line like that, you can instead assign any module to any group. The group is just a logical name but it allows you to load or unload whole module groups. This is nice because your code does not need to know what’s in the group, just that you want to load or unload it. This addressed the nasty coupling that I was seeing in the scripts. Being able to have your application load or unload a logical set of modules is very nice indeed. Also, internally, modules are reference counted so if a module needs loading but it’s already loaded then it gets a reference count bumped-up. Likewise for unloading where a module will only be unloaded if it has no references.

Another extremely useful feature is dependencies. You can define which modules this module depends on, even specify a specific version. This informs the module system that to load this module requires that the specified modules need to be loaded first. This happens recursively and can automatically detect and warn about cyclic dependencies. It can also warn about incompatible versions. The only rule with the module manager is that a only a single version of any module id can be loaded at once. This is because a new version is a breaking-change of the previous version but also has potentially conflicting function and object names or behavior. Each module does have a serial-Id and the asset system has methods that allow you to update modules for publishing. The scripts that presented that to the end-user is in 3-Step studio but the infrastructure for it is in the module manager.

This dependency and version might seem like overkill for some and you are free to just leave all your modules at version “1” and be done with it and that’s fine. However, there’s a whole project synchronization infrastructure designed around modules that allows very cool things like having a game project that can have modules updated after the fact. Even allowing new modules to come along and not replace existing modules being used in a game project. All this without requiring that a game project have copies of all the module code placed into the project folder. Essentially the code just asks for modules and the engine takes care of the rest.

One final feature is that the “^” expando character used when expanding and collapsing paths is now used a lot in T2D. This is because when a module is loaded, an expando is automatically generated with the module-Id. This means that for any module, code within it can get its own location by using something like:

TamlWrite( “^MyModule/scene.taml”, %scene );

This will write the file “scene.taml” in the root folder of the “MyModule” location.

Asset System

Gone are the days of a single script file containing, typically, a SimSet that contained a bunch of datablocks containing state that defined some kind of asset. Most of the time this was something like the t2dImageMapDatablock and always each used the Torque object-name feature where an object was assigned a global name that could conflict with other objects. Goodbye, I won’t miss you.

Like the module system, the asset system is a complex system to describe and there’s a huge internal wiki document that does just that however the day to day use of it is crazy simple. Most of the asset system functionality is for an editor as it provides lots of administration functionality for generating, destroying and querying assets and asset meta-data.

So previously you had a file containing TorqueScript code that described an object. Each of these objects contained all the fields for that object. Now, that file has been replaced with an asset definition. An asset definition describes an asset but in reality, it’s just an object persisted using Taml.

The legacy T2D didn’t have any real definition of an asset, it was just a term used to mean “something that objects consumed”. In the new T2D, there are real assets and they are all derived from a type named “AssetBase”.

The initial set of assets available are:

• ImageAsset (Replaces the old t2dImageMapDatablock)
• AnimationAsset (Replaces the old t2dAnimationDatablock)
• AudioAsset (Replaces the old AudioProfile)
• ParticleAsset (Replaces the old t2dParticleEffect & t2dParticleEmitter)

The asset system provides a lot of functionality but essentially it allows you to scan a folder for assets. When these are discovered it maintains an internal database of them all. They can then be used as soon as they are discovered.

Here’s an example of a basic ImageAsset type:

<ImageAsset
    AssetName="background"
    ImageFile="@assetFile=#background.png" />

This is an asset named “background” that uses the “background.png”. The “#” you see is a shortcut meaning a file relative to the location of the asset definition file. You can use anything you like here, even relative locations but it’s common for the asset definition to be located alongside the asset data.

Now it may scare you that at start-up the system is going to scan and load all assets but that absolutely isn’t the case. What does happen is that the asset system knows about the asset definitions and their names but does not load the assets, that’s done elsewhere when required.

An important aspect of assets is that they have a partially-qualified and fully quality identity. In the example above, the asset is named “background” and this is the partially-qualified name. The thing is, you will NEVER refer to this asset as background and the reason is that it’s quite likely that you’ll conflict with another asset named “background”. If you keep tight control over your asset names then that might not happen though. Indeed, historically folks used horrible name encodings to try to make each unique.

The asset system gets around this however forcing assets to be contained within a module. Modules not just for code, they can contain assets. Indeed, modules can be thought of as isolated “asset packs” which you can obviously load/unload, even have versions for them! This means that if you were to get an “asset pack” module from a buddy or from someone selling one and want to integrate it within your game, there’s a good chance that it’ll contain assets with names that conflict with ones you already have.

To get around this, assets are always referred to by their fully qualified names. This is in the form of:

ModuleId : AssetName

So for instance, in a module Id of “EnemyAssets” with an asset named “Monster” you’d assign it to a sprite like this:

%sprite.Image = “EnemyAssets:Monster”;

This fully-qualified form is known as the asset-Id and it’s what you’ll be using all over the place. This means that you can have two assets named “background” and it won’t matter because they must be in different modules. If you attempt to use assets of the same name within the same module then the asset system will warn you about it, giving you a lot of detail about what went wrong.

This configuration is very nice because as an author of these assets, you have the freedom to both configure and name your assets however you like. You can use whatever naming conventions you so choose as well knowing that the asset-Id is generated by prefixing the module-id of your asset pack and that it won’t conflict in any way with other modules. The trick of course is ensuring that your module-id is unique but it’s a nice high-level scope that everyone will use.

Also, given the example above, the assets in the module “EnemyAssets” won’t become available until that module is loaded. If you perform queries on the asset manager, you won’t see these assets if the module is not loaded. This confirms that the system doesn’t scan and load all assets at start-up.

Another huge change with this system is how and when assets are loaded. When you reference an asset, it is acquired and assigned automatically. If you assign the same asset to another object that uses it then it gets it immediately without the asset being loaded (it’s already loaded). In other words, assets are reference counted, The same goes for when assets are no longer referenced, they are unloaded although you can control the unloading allowing all assets or specific ones to stay in memory until the module is unloaded or your perform an asset purge.

Unloading of assets might have veteran users of legacy T2D worried. The old t2dImageMapDatablock was powerful in that it could take any sized image and ensure that it was packed into a texture (or textures) that were compatible with the target hardware. That’s great for novices however it’s a content-pipeline process that is being executed at runtime and was the number one problem on mobile platforms causing considerable performance problems.

The new ImageAsset does none of that. Gone are the crazy modes and a few dozen options. The image that is loaded is the texture that gets uploaded to the GPU. You can section the texture as before into frames but it’s a simple load the thing up and get it as a texture rather than the legacy load up the image, copy parts of the image and pack them into a memory bitmap (or bitmaps) then upload them to the GPU. it also means that things like resolution switching on desktop platforms is super-quick as it does not require a whole bunch of stuff being calculated.

This simplified configuration and the use of Taml in Xml format means that you can consume XML based data configuring things like image-assets. This is why when I downloaded the demo of TexturePacker and looked at how it can be customized to output a custom file it took 20 minutes to get it to work. It actually took longer to grab this video showing me doing it:



For C++ developers, there’s an internal smart-pointer system that allows you to reference assets without any concern over how they are loaded or even persistence. Even Taml knows about assets (not directly but indirectly via the smart asset pointer) meaning you can have fields that expose the asset assignment and they persist using Taml.

Without going into more detail of each feature I’ll highlight some other awesome features of the asset system as well. For one thing, T2D “knows” about the assets you use in a game project. As I said, I won’t go into the detail here but the great thing is that the system is aware of what assets your project is using and therefore perform publishing options like only publishing the assets you use. It therefore knows about assets being used that don’t exist and can inform you of that too. It not only knows what assets you use but it also knows where you use them. This means that if you were to ask the asset system to rename an asset then it knows to rename the asset references you have. Also, it can delete assets for you and it knows not only where they are used but also what “loose files” are being used. A loose file is something like an “audio file” for an Audio Asset or an “image file” for an Image Asset etc. This is known as asset referencing and loose-file referencing.

T2D not only supports asset referencing from the obvious things like scene-objects but it also knows about references of assets from other assets i.e. assets using other assets. This may sounds strange but it’s actually quite common. Take, for instance, an AnimationAsset. That might be defined like this:

<AnimationAsset
    AssetName="BananaWiggle"
    Image="FruitAssets:Banana"
    AnimationFrames="0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15"
    AnimationTime="5" />

This is an animation asset and as you can see, it has a field of “Image” which is an asset reference, in this case an “ImageAsset” reference. When this “BananaWiggle” asset is discovered by T2D it not only knows about this asset but it also takes note that it requires i.e. depends on another asset of “FruitAssets:Banana” of the type “ImageAsset”.

This means that the animation asset requires an image asset. It also means that if you assign the animation to a sprite like this:

%sprite.Animation = “FruitAssets:BananaWiggle”;

… then the asset system will first load the “FruitAssets:Banana” image-asset (assuming it’s not already loaded) followed by the animation. Of course if any of these things are already loaded then nothing is loaded, just reference counted. The same goes, of course, if they are not reference any more such as destroying a sprite that used the animation. In this case both the animation and the image assets have their references reduced which may result in them being unloaded.

Although primarily for an editor, the asset system provides a dedicated query system that allows you to perform asset queries on all the asset meta-data. You can even pass previous queries to refine results. So for instance you can query all assets of a particular asset type i.e. "ImageAsset" then refine the search for assets that have the word "Zombie" in them and continue like that. The system uses a dedicated AssetQuery object to completely remove the need to pass long and slow strings back and forth. When you've finished performing your query or queries you can retrieve your results in very much the same way you would retrieve objects from a SimSet. Asset queries return asset-Ids which allow you to perform work on your results.

A final thing I’ll mentioned about the asset system is its notification system. If an asset changes, in any way whatsoever, then anything that uses it gets notified if it wants to. Assets don’t change often and nearly always in an editor so there’s very little to worry about in a game runtime nevertheless, it’s still super-fast as the notification is a simple callback. The net result is that any object using an asset knows that it’s been updated and respond accordingly. This also applies to other assets that may be using another asset i.e. an animation uses an an Image asset. If an image asset was updated then any animation assets using it would also be updated and their frames validated. Anything using those animations like a scene sprite or a GUI control would see the change instantly.

Whilst this is used typically in an editor, you are free to use it for cool things in your game should you so choose.

There’s a huge set of features here but far too much to go into now. It’s safe to say that the asset system is powerful but it’s also really easy to use. You just assign the asset-id and all is done for you.

These two systems are by far the biggest change to T2D and at first they are a lot to take in but then you realize that actually using them is really simple. In an attempt to show you not only what the new Sandbox set-up looks like but also working with modules and assets, here's a video of me using a simple toy to create a sprite using a static asset rotating on the screen:



Part 1 of this blog can be found here.
Part 3 of this blog can be found here.