Torque 3D Open Source Update: Template Refactor
by Michael Hall · 11/30/2012 (1:05 pm) · 13 comments
Planning discussion
For utmost flexibility it is my belief that the Template(s) should be a minimal and genre-agnostic framework for the end-user to build upon. Unfortunately, as it currently exists, the Full Template contains some FPS-centric elements, and other bloat, that clouds the issue of just what is necessary for a minimal script base. Removing these extraneous features will help establish an initial effort in a longer term goal of refining and refactoring our basic Template offering, making it cleaner as well as easier to support the evolving needs of the community.I submitted a detailed outline to the Steering Committee of what I was wanting to remove or modify for their thoughts in refining a course of action, which in summary could be outlined like so:
- Remove unused/legacy global variables
- Correct the cause of some console warnings and/or errors
- Asset cleanup to avoid duplication
- Remove the FPS specific features
- Reduce complexity of stock scripts
- Directory restructuring
From that list of actions I proposed a plan for a "starter" template that genre specific Templates such as FPS Template, RTS Template, etc., could be built from.
The Starter Template would contain the script necessary to:
- Initialize all sub-systems
- Demonstrate a full but basic user interface (console, main menu, level select, server join/host, options, etc.)
- Load a level
- Provide an example shape and required datablock/scripting for each object class.
- Demonstrate how to drop a player into a mission in either in 1st or 3rd person perspective.
- Demonstrate minimal gameplay rules for ending the game or advancing to another level. Additional gameplay aspects would be specific to the various Genre Templates.
- Provide support for the GUI and World Editor tools.
- Retain standard package and/or mod support
Additionally, but of lesser of importance, would be an even more stripped down template; a "Barebones Template" that would contain the minimal script necessary to:
- Initialize all sub-systems
- Provide access to the console
- Provide support for the GUI and World Editor tools
Workflow and Progress Report
Initially I had organized three different template projects (FPS, Starter, Barebones) that I was working on at once. This proved tedious.To shortcut the whole process I re-purposed the current Full Template to become the FPS Template, which will then be stripped down even further to become the Starter Template. This more easily prevents having to work across several different templates at once, performing the same steps over and over within the duplicate tools and core directories.
And that is where I am at now. There may still be some minor fixes and cleanup that could be implemented, but the bulk of the work is done and I am ready to begin the directory restructuring phase. It is at this point that it was suggested that we approach the community for feedback in regards to the plan for this directory restructuring.
Community Interaction
What this means in consideration of my plan will be the merging of the "core" and "game" packages. This merge would combine both sets of scripts and assets reducing a layer of complexity -- with no loss of functionality! We have yet to decide just how to setup the new structure, but basically it could go one of two ways:- Move the current "game" mod (art, levels, scripts) into the baseline"core" module.
- Explode the "core" out and into the "game" mod package.
I don't really favor one direction over the other. Moving things into the core would be the easiest to do since there are texture and gui assets that are hardcoded in the source to be found in the core/art directory. But there is a goal that we are investigating to remove these hardcoded filepaths. If this is accomplished, then that would make it much easier for anyone to implement a directory redesign of their own.
One of the biggest concerns that arose in our discussion was that significantly altering the directory structure would in essence break the instructions for every existing Resource, tutorial, content pack installation, etc. Third party content developers and tutorial writers are going to hate me for that!
So this the question that we pose to you, the community: do you want to see backwards compatibility maintained, or would you rather see the Template broken down and rebuilt from the ground up to be more flexible -- and easier to use -- for end-user and future development? If the latter, are there any suggestions about the direction of the core / game merge?
Moving Forward
Keep in mind that this is not to be taken as a promise of a course of action, only as an indication of the possibilities we are investigating in the committees' ongoing effort to streamline and codify a working roadmap. Each one of us in the steering committee has expressed a desire to maintain transparency to the community. Knowing more about what you all want, or even just how it makes you feel, from the various things we are working on helps us better evaluate the direction that is most beneficial.About the author
Been dabbling with game-programming since the age of 10 when I got my first computer, a Commodore. Got serious about game-development after modding Tribes for several years. Doesn't sleep much. Drinks rum. Teaches guitar. Plays cello.
#2
IMO, backwards compatibility is secondary to producing a fantastic product. But I probably do underestimate the number of headaches that could be caused by refactoring the directory structure too severely!
Whichever direction is taken, I think the most important thing for templates is that there be no repetition. If I write a resource or submit a pull request that modifies scripts, I don't want to have to touch the same file in three different places! There should be one set of editor scripts, one core directory, etc.
Also, nice blog image ;).
11/30/2012 (2:46 pm)
Really excited for this. I wouldn't be opposed to seeing most functionality moved into the core directory, with the game directory being for, as the name suggests, game-specific code.IMO, backwards compatibility is secondary to producing a fantastic product. But I probably do underestimate the number of headaches that could be caused by refactoring the directory structure too severely!
Whichever direction is taken, I think the most important thing for templates is that there be no repetition. If I write a resource or submit a pull request that modifies scripts, I don't want to have to touch the same file in three different places! There should be one set of editor scripts, one core directory, etc.
Also, nice blog image ;).
#3
11/30/2012 (5:10 pm)
Disable all hard coding and go with option 2: Explode the "core" out and into the "game" mod package. so there is just the game/subfolders and no game/core folder. When your whole game becomes the "core" then there's no point in having a seperate "core" folder.
#4
11/30/2012 (5:26 pm)
I love the idea of a starter template! In addition one thing I was thinking that may help would be to add gerne specific templates like an RTS, FPS, 3rd person, racing, etc, etc. That way people would still have those templates to learn from but have the starter just in case they would rather build gameplay elements themselves or maybe those could be resources like Daniel's AI behavior manager work and also be put on the GIThub so that people could download according to the project they are working on.
#5
12/01/2012 (5:16 am)
I'm with Steve on this one,Quote:
Disable all hard coding and go with option 2: Explode the "core" out and into the "game" mod package. so there is just the game/subfolders and no game/core folder. When your whole game becomes the "core" then there's no point in having a seperate "core" folder.
#6
The template restructuring will not be in the next master branch update, but we are aiming for the one after that. Also, even with the removal of hardcoded filepaths in the source there shouldn't be anything preventing the use of the existing Templates, other than the need for the user to update them in regards to possible script fixes.
@Daniel: I'm actually wanting to get away from the "core" + "game" split. It confuses newcomers, and adds an extra layer of obfuscation when you find yourself wondering where a function is being overridden. Merging them together, and then modifying each Template towards a specific, yet generic, game type would make it easier to facilitate direct improvements to a FPS Template, Racing Template, RTS Template, etc.
EDIT: with that said, I am planning for a sub-directory (most likely under scripts/client) that will contain the bulk of the "core" functionality that doesn't affect gameplay. The init'ing of subsystems and such -- the things that actually don't change from game to game.
It's a little known bit of knowledge, but it only takes just a little modification to make the game start up with nothing but the "core" package. Which is great, but it has annoyed me for years that there is so much game specific script tucked away in the core. If we were to maintain that split, it further complicates the issue of duplication -- especially given how often I find myself needing to override things in the core, which varies from game type to game type. That is why the Starter Template will be as generic as we can get away with. From such an already stripped down starting point it would be easier to build upwards to specific GameType Templates without the need of always stripping the FPS-centric stuff out.
RE: duplication:
Duplication of some scripts and GUIs will be unavoidable -- no matter how things are arranged or setup. We are looking at ways to reduce the duplication of specific directories in the repository; tools being a good example. Unfortunately git doesn't handle SVN type externals quite as easily as hoped -- it introduces extra complexity to checking out that we feel would be discouraging for some, so that option is looking less and less likely. But we think that the project generator could be tweaked such that it could pull script modules from a "common" or "shared" location when a project is generated. This would mean that the Templates themselves wouldn't work out of the box, but when using the Project Generator, the project would be created by pulling from the chosen Template, copying the tools into the project, and pulling the necessary dlls if physX is enabled, etc. No decision has been made on this though, as it's still being discussed and researched.
@Kory: currently our only immediate planned genre Template is the FPS Template -- simply because that's already what the Full Template is, and would help retain a sense of familiarty. As I already mentioned, the Starter Template layout should definitely make it much easier to build other genre Templates, which inevitably will appear in time.
@Steve & David: My game is the "core", not a mod! :)
Since it's looking like there is no obvious outcry against this plan, I'll be moving on to implementing the restructuring as per option 2, unless more feedback or suggestions appear which would obligate further discussion.
I did have thoughts as to how we could leverage option 1 to allow for easy drop-in usage of art and code packages (treated as mods), but that would likely complicate things for third party developers and force them to adhere to a specific design principle... which technically would still be possible with option 2, just not as pretty.
12/01/2012 (2:07 pm)
@Jimmy: Since it wouldn't make sense for us to maintain multiple versions moving forward, any previous version from each release is technically locked and frozen. Each update of the master branch should have a version id, as well as a release download -- there will be more about that later. No one is forced to use the development branch, but it is a good indication of how things are transitioning in regards to next master branch update.The template restructuring will not be in the next master branch update, but we are aiming for the one after that. Also, even with the removal of hardcoded filepaths in the source there shouldn't be anything preventing the use of the existing Templates, other than the need for the user to update them in regards to possible script fixes.
@Daniel: I'm actually wanting to get away from the "core" + "game" split. It confuses newcomers, and adds an extra layer of obfuscation when you find yourself wondering where a function is being overridden. Merging them together, and then modifying each Template towards a specific, yet generic, game type would make it easier to facilitate direct improvements to a FPS Template, Racing Template, RTS Template, etc.
EDIT: with that said, I am planning for a sub-directory (most likely under scripts/client) that will contain the bulk of the "core" functionality that doesn't affect gameplay. The init'ing of subsystems and such -- the things that actually don't change from game to game.
It's a little known bit of knowledge, but it only takes just a little modification to make the game start up with nothing but the "core" package. Which is great, but it has annoyed me for years that there is so much game specific script tucked away in the core. If we were to maintain that split, it further complicates the issue of duplication -- especially given how often I find myself needing to override things in the core, which varies from game type to game type. That is why the Starter Template will be as generic as we can get away with. From such an already stripped down starting point it would be easier to build upwards to specific GameType Templates without the need of always stripping the FPS-centric stuff out.
RE: duplication:
Duplication of some scripts and GUIs will be unavoidable -- no matter how things are arranged or setup. We are looking at ways to reduce the duplication of specific directories in the repository; tools being a good example. Unfortunately git doesn't handle SVN type externals quite as easily as hoped -- it introduces extra complexity to checking out that we feel would be discouraging for some, so that option is looking less and less likely. But we think that the project generator could be tweaked such that it could pull script modules from a "common" or "shared" location when a project is generated. This would mean that the Templates themselves wouldn't work out of the box, but when using the Project Generator, the project would be created by pulling from the chosen Template, copying the tools into the project, and pulling the necessary dlls if physX is enabled, etc. No decision has been made on this though, as it's still being discussed and researched.
@Kory: currently our only immediate planned genre Template is the FPS Template -- simply because that's already what the Full Template is, and would help retain a sense of familiarty. As I already mentioned, the Starter Template layout should definitely make it much easier to build other genre Templates, which inevitably will appear in time.
@Steve & David: My game is the "core", not a mod! :)
Since it's looking like there is no obvious outcry against this plan, I'll be moving on to implementing the restructuring as per option 2, unless more feedback or suggestions appear which would obligate further discussion.
I did have thoughts as to how we could leverage option 1 to allow for easy drop-in usage of art and code packages (treated as mods), but that would likely complicate things for third party developers and force them to adhere to a specific design principle... which technically would still be possible with option 2, just not as pretty.
#7
12/03/2012 (8:48 am)
+1 to remove "core" and move all that stuff into "game". Like Steve said, the game IS the core, and shouldn't be treated as a mod.
#8
12/03/2012 (12:56 pm)
Get rid of "packages". They are really only useful for different game types and not large chunks of code. I don't like them because they make the code hard to figure which function is being called. Maybe this is just me, but I don't like packages.
#9
The mission based game-types will remain in the FPS Template, but I'll be removing that functionality from the Starter Template. We may take a closer look at removing the "game" package in the Barebones Template.
12/03/2012 (1:20 pm)
@Frank: The number of package levels in the project will be reduced in order to avoid confusion over which function is being called -- no more forcing the user to override the core. The benefit of retaining at least an encapsulating package outweighs that of removal.The mission based game-types will remain in the FPS Template, but I'll be removing that functionality from the Starter Template. We may take a closer look at removing the "game" package in the Barebones Template.
#10
12/03/2012 (1:34 pm)
Not sure if I'm allowed to have an opinion here considering my employer (cough). Hopefully Dave doesn't mind, but my approach would be to blow the core up. Why? We just went through the same process with T2D. Over the years, "core" and "common" have become synonymous with "dumping ground". Crap gets thrown in there because the dev knows it will be loaded, because it's "core". Just my 2 cents.
#11
It's that "dumping ground" approach over the years which prompted my desire to want to restructure the Template(s).
12/03/2012 (1:47 pm)
@Michael: All opinions are welcome, even... employees ;)It's that "dumping ground" approach over the years which prompted my desire to want to restructure the Template(s).
#12
I went hunting for the screenshot functionality the other day... surprise, surprise, it was in core! Except for HUD-less screenshot, that was in the game scripts.
12/05/2012 (7:19 pm)
Yeah, not sure what I was thinking when I made my last post. The idea of a script 'core' sounds like it should be restricted to the very basics required to get Torque to run, and not much else. It could even be a handful of script files, instead of an entire mirror of the game's directory structure.I went hunting for the screenshot functionality the other day... surprise, surprise, it was in core! Except for HUD-less screenshot, that was in the game scripts.
#13
Way back when, GG had the core set as external on their SVN repository, so it got duplicated across all projects. That played a large part in why so much got dumped into the core over the years -- it's easier and quicker to make one change than it is to make that change across several different (slightly) projects. And that makes sense for in-house development of demos and examples, but leads to the current problem of confusion and bloat that we see today.
The hudless screenshot is something I added for the FPS Genre Kit, and wasn't something that was to be reflected across all projects so I added as "game" code. I really wanted to start moving things out of the core back then...
It was always my belief that with the advent of the project generator that the core should be the template, instead of trying to maintain the core + mod approach. Once you have a starting base template you should just be able to build a new project and add what you want to it, instead of overriding and extending a core with changes to an example mod. So hopefully with the new restructuring direction we're taking we can avoid that type of confusion and frustration.
12/07/2012 (1:41 pm)
I totally understand the frustration of hunting for something in the scripts and not knowing if it's part of the "core" or is "game" code.Way back when, GG had the core set as external on their SVN repository, so it got duplicated across all projects. That played a large part in why so much got dumped into the core over the years -- it's easier and quicker to make one change than it is to make that change across several different (slightly) projects. And that makes sense for in-house development of demos and examples, but leads to the current problem of confusion and bloat that we see today.
The hudless screenshot is something I added for the FPS Genre Kit, and wasn't something that was to be reflected across all projects so I added as "game" code. I really wanted to start moving things out of the core back then...
It was always my belief that with the advent of the project generator that the core should be the template, instead of trying to maintain the core + mod approach. Once you have a starting base template you should just be able to build a new project and add what you want to it, instead of overriding and extending a core with changes to an example mod. So hopefully with the new restructuring direction we're taking we can avoid that type of confusion and frustration.

Jimmy R Armes
another possible solution is to keep the current 1.2 as is and freeze it in it's development and then make all the necessary changes in a brand new realase giving it a new version number all together. Then let people decide to use what ever engine they feel would give them what they needed.