Torque 3D Open Source Update: Template Refactor #2
by Michael Hall · 12/07/2012 (2:56 pm) · 14 comments
As announced in the Torque 3D Open Source Update: Template Refactor blog, we created a design plan that would involve cleanup and restructuring of the Template projects, and approached the community in an effort to let you provide feedback and help guide the design for the final plan. Since this is something that will affect following instructions for all guides, resources, and content pack installation we are wanting to maintain this transparency by keeping the community informed.
Based upon the responses received I have begun the refactoring process such that the core directory will be removed. On the surface this involves merging the scripts, GUIs, assets, and data found within the core module into the user's game directories. Furthermore there will be a few changes in regards to how things are organized within the new Templates.
This will result in a directory structure with the following sub-directories relative to the game directory:
You'll notice that the images sub-directory is new here. It will contain all of the images or textures for game objects that are not associated with a shape. This will be the various default / missing / warning textures, billboard foliage, particles, lensflares, decals, road & path textures, etc. This will make it easier to find specific asset types by navigating to the appropriate location, instead of swimming through the previous just-dump-things-in-there approach taken with the old art directory.
Note that the gui sub-directory has been removed from the art directory. More about this change later.
The data/server directory will contain the following:
An interesting problem that we face with the GUI profiles is that the tools share many of the profiles and images with those found within the core. What I am initially going to do here is to simply duplicate those that are shared with the tools within the tools module itself. The plan is that at some future date, time and resources permitting, the look and feel of the various editors could more easily be modified without affecting the rest of the game interfaces.
Instead of hunting for the various locations that you can exec additional scripts, GUIs, etc., from I will be tweaking the exec order a little so that there are individual and discrete locations for executing various aspects of Torque. One location for datablocks, one for script, and one for GUIs. It's little understood, but with the exception of a few key files or objects, and unavoidable datablock dependencies, Torque is very flexible with when and where you exec most files.
Argument parsing for the command line also seems a bit bloated, anyone have any thoughts or concerns on minimizing our default command line options?
Also, keep in mind that once the hard-coded filepaths in the source are removed, it will be much easier to implement a directory structure of your own. However, there will by necessity be some hard-coded references in the scripts and GUIs -- but script is much easier to change at will than engine code is. The biggest crossover here will be the filepaths to GUI assets, which is why the choice was made to try and keep every thing GUI related in one location.
Based upon the responses received I have begun the refactoring process such that the core directory will be removed. On the surface this involves merging the scripts, GUIs, assets, and data found within the core module into the user's game directories. Furthermore there will be a few changes in regards to how things are organized within the new Templates.
This will result in a directory structure with the following sub-directories relative to the game directory:
- art
- data
- gui
- scripts
- shaders
- tools
- web
ART
The art directory has been cleaned up, removing unused and duplicate assets. Directly under art will be found three sub-directories:- shapes
- sounds
- images
You'll notice that the images sub-directory is new here. It will contain all of the images or textures for game objects that are not associated with a shape. This will be the various default / missing / warning textures, billboard foliage, particles, lensflares, decals, road & path textures, etc. This will make it easier to find specific asset types by navigating to the appropriate location, instead of swimming through the previous just-dump-things-in-there approach taken with the old art directory.
Note that the gui sub-directory has been removed from the art directory. More about this change later.
DATA
Essentially this will contain datablocks, levels, and other data that the engine reads and/or exports that is not art or contains no script functions. The various *.prefs.cs, config.cs, banlist.cs, etc files are examples of this other kind of data. We are also wanting to distinguish between server-only and client-side data. To this effect there will be a server sub-directory and a client sub-directory located here.The data/server directory will contain the following:
- datablocks
- levels
- managed - this will be the default location for data saved from the various editors.
- audioprofiles
- gfxprofiles
- prefs
GUI
The gui directory will be a combination of all of the .gui files, GUI specific scripts, and GUI image assets, all conveniently located in one place. The fonts cache sub-directory will also be located here.An interesting problem that we face with the GUI profiles is that the tools share many of the profiles and images with those found within the core. What I am initially going to do here is to simply duplicate those that are shared with the tools within the tools module itself. The plan is that at some future date, time and resources permitting, the look and feel of the various editors could more easily be modified without affecting the rest of the game interfaces.
SCRIPTS
Scripts will maintain the current server and client-side organizational split. The only difference here will be the merge of the core and game packages.SHADERS - TOOLS - WEB
There will be no changes to these directories at this time.OTHER THINGS OF NOTE OR RELEVANCE
There was mention of confusion and/or dislike of script packages within Torque. This will remain, although reduced in apparent complexity with fewer overridden function calls. I feel that there are far too many benefits to keeping this functionality intact than there are in removing it. There is a lot of power and flexibility in using script packages once you understand how they work.Instead of hunting for the various locations that you can exec additional scripts, GUIs, etc., from I will be tweaking the exec order a little so that there are individual and discrete locations for executing various aspects of Torque. One location for datablocks, one for script, and one for GUIs. It's little understood, but with the exception of a few key files or objects, and unavoidable datablock dependencies, Torque is very flexible with when and where you exec most files.
Argument parsing for the command line also seems a bit bloated, anyone have any thoughts or concerns on minimizing our default command line options?
Also, keep in mind that once the hard-coded filepaths in the source are removed, it will be much easier to implement a directory structure of your own. However, there will by necessity be some hard-coded references in the scripts and GUIs -- but script is much easier to change at will than engine code is. The biggest crossover here will be the filepaths to GUI assets, which is why the choice was made to try and keep every thing GUI related in one location.
About the author
Been dabbling with game-programming since the age of 10 when I got my first computer, a Commodore. Got serious about game-development after modding Tribes for several years. Doesn't sleep much. Drinks rum. Teaches guitar. Plays cello.
#3
12/07/2012 (6:53 pm)
This sounds great. One thing with packages that would be good. Have them in one location if possible. Maybe even have a set of files called package scripts. So it is relatively easy to look for the source files for those. Then for the functions that get overriden have a note that this function is intended to be overriden. I think the frustrating part was finding a stub function and not knowing where the overriden function is defined. It is a good feature especially for game types, but it is just a pain to find where the functions are overriden. 
#4
12/07/2012 (7:59 pm)
Nice work. Something I just remembered - can particles be put in 'images', not 'shapes'? I've never understood the why of that particular decision. 
#5
12/07/2012 (8:52 pm)
This all sounds good, but does this mean that the new 3D Programming All in One 3rd edition is already out of date after being released for less than a week? LOL. My copy hasn't arrived yet, but I previewed the Ebook version and I can just laugh at the concept of first timers yelling, "The files weren't in that directory!" 
#6
12/08/2012 (7:37 am)
Once upon a time that book shipped with a version of Torque (and its directory structure was modified from stock). If this is still the case then there should be fewer issues.
#7
@Frank: I gather from some of your recent posts that you don't like packages due to that functionality not being easily portable to other languages. Coming from a background as a Tribes modder I'll admit to having a tendency of dismissing some concerns for aspects of Torque script that others quibble over -- but I'm working on that!
For packages, I think the largest stumbling block for anyone is the whole core + gameMod approach. Since I'll be merging those two packages, the confusion factor should be much less. There will be one encapsulating package, the game. An individual's specific project needs could dictate the removal of the wrapper package altogether, but for now it stays since we've got to look at it as being of benefit and flexible for general use. I have however, planned from the top down to try and reduce the amount of possible confusion due to use of packages. The FPS Template will have minimal, if no, duplication or overriding of formerly "core" methods, since with the merge of the core there goes the need for some extended code -- you just write that functionality into the standard package. The gametype object functionality will continue to use packages for each mode or type. The Starter Template will only have the one game package. It's yet to be decided upon, but the Barebones Template will possibly not make use of package functionality at all.
@Daniel: Thanks! Particle images being located under shapes is just one of the illogical things I pointed out to the rest of the committee. The work to move them to art/images/particles has already been done :)
@William: Yes, that is certainly likely to happen. That's why we want to keep this restructuring transparent to everyone. But since we're are not modifying the current templates, only replacing them with other examples, the impact of that scenario should not be as severe. It will be some time yet before the new templates are pushed to the Github repository. Only after the upcoming release, a push to the master branch, will the new templates be added to the development branch. So we wont be forcing this change on anyone out of nowhere.
12/08/2012 (10:44 am)
@Steve & Kory: thanks guys! I was hoping that others than me would find the proposed layout sensible. If not, it's their time to speak up.@Frank: I gather from some of your recent posts that you don't like packages due to that functionality not being easily portable to other languages. Coming from a background as a Tribes modder I'll admit to having a tendency of dismissing some concerns for aspects of Torque script that others quibble over -- but I'm working on that!
For packages, I think the largest stumbling block for anyone is the whole core + gameMod approach. Since I'll be merging those two packages, the confusion factor should be much less. There will be one encapsulating package, the game. An individual's specific project needs could dictate the removal of the wrapper package altogether, but for now it stays since we've got to look at it as being of benefit and flexible for general use. I have however, planned from the top down to try and reduce the amount of possible confusion due to use of packages. The FPS Template will have minimal, if no, duplication or overriding of formerly "core" methods, since with the merge of the core there goes the need for some extended code -- you just write that functionality into the standard package. The gametype object functionality will continue to use packages for each mode or type. The Starter Template will only have the one game package. It's yet to be decided upon, but the Barebones Template will possibly not make use of package functionality at all.
@Daniel: Thanks! Particle images being located under shapes is just one of the illogical things I pointed out to the rest of the committee. The work to move them to art/images/particles has already been done :)
@William: Yes, that is certainly likely to happen. That's why we want to keep this restructuring transparent to everyone. But since we're are not modifying the current templates, only replacing them with other examples, the impact of that scenario should not be as severe. It will be some time yet before the new templates are pushed to the Github repository. Only after the upcoming release, a push to the master branch, will the new templates be added to the development branch. So we wont be forcing this change on anyone out of nowhere.
Quote:
One of the biggest concerns that arose in our discussion was that significantly altering the directory structure would in essence break the instructions for every existing Resource, tutorial, content pack installation, etc. Third party content developers and tutorial writers are going to hate me for that!
#8
Levels are one contrary PITA! It was decided to categorize levels, or missions, as server data. Which is sound in principle, but leaves me with the conundrum of catering to the needs of the terrain. Terrain textures were moved into art/images/terrains -- simple enough. But what to do about the terrain file itself, which is generated from the terrain editor? Is the .ter file to be considered data, or an asset? Wherever the terrain file resides is also where the basetex.dds terrain texture will be generated. Then there are postFX settings, mission lighting under basic lighting (.ml files), and .mis.decals that all get applied to individual levels as well. So there we go mixing various amounts of server and client data.
We may have to rethink the server / client organization of the data directory...
12/08/2012 (11:07 am)
LevelsLevels are one contrary PITA! It was decided to categorize levels, or missions, as server data. Which is sound in principle, but leaves me with the conundrum of catering to the needs of the terrain. Terrain textures were moved into art/images/terrains -- simple enough. But what to do about the terrain file itself, which is generated from the terrain editor? Is the .ter file to be considered data, or an asset? Wherever the terrain file resides is also where the basetex.dds terrain texture will be generated. Then there are postFX settings, mission lighting under basic lighting (.ml files), and .mis.decals that all get applied to individual levels as well. So there we go mixing various amounts of server and client data.
We may have to rethink the server / client organization of the data directory...
#9
12/08/2012 (2:20 pm)
It would be much better if Client and Server were separate. Ideal for single player games, multiplayer and MMOs. Less work involved for the end developer, and smaller packages for the end result.
#10
12/09/2012 (10:54 am)
@Jules: that's certainly an option that could have been investigated, but we made the structural decision based upon the feedback received as to what was most desirable to the community. However, it will be much easier for the end user to make that separation on their end than it was before. 
#11
12/13/2012 (1:35 pm)
Thanks for this Michael. Seems so much more logical than rummaging around in core/. 
#13
main.cs
A simple Torque 3D window that does absolutely nothing.
03/07/2013 (1:06 am)
Quote:How about a single script file:
How about a template that only opens a simple canvas?
main.cs
// Set the name of our application
$appName = "Base Template";
enableWinConsole(true); // optional, until console.gui is ready
new GuiControlProfile(GuiDefaultProfile); // need to fill this in
new GuiControlProfile(GuiToolTipProfile); // this too
new GuiCanvas(Canvas);
Canvas.setWindowTitle("T3D - "@ $appName);
//exec("sys/modules.cs"); // load the things that the client will need (audio, postfx, lighting, etc
new RenderPassManager(DiffuseRenderPassManager);
setLightManager("Basic Lighting");
activateDirectInput();
GlobalActionMap.bindCmd("keyboard", "escape", "quit();", ""); // just to make it easy to quit.A simple Torque 3D window that does absolutely nothing.
#14
03/07/2013 (1:12 am)
The template restructuring has been set aside in favor of a ground up rewrite that heavily favors modularity in design and function. This will not be ready in time for QA before the next release. More information in my next (soonish) blog update. 
Associate Steve Acaster
[YorkshireRifles.com]