Pluggable terrain engine
by Brian Hook · in Torque Game Engine · 02/28/2003 (9:03 pm) · 36 replies
The existing terrain engine has some well documented limitations that I know others are working around, however the technology is decidedly "old-school". Newer terrain rendering technology exists (e.g. Ulrich's "Chunked LOD") that is, by and large, better for most modern computers.
Dropping in a new terrain engine is, obviously, non-trivial, but I don't believe it is fundamentally insurmountable. From an engine standpoint (not tools) it looks like the requirements would be to abstract out the fundamental terrain representation and remove any existing implementation dependencies from the game body itself.
Ideally things like WaterBlock::mCameraSubmerged would instead become effects flags, thereby decoupling the game or client from knowing about WaterBlocks directly. Material properties would be queried indirectly instead of referenced directly from the TerrainBlock, etc.
Once that basic infrastructure was in place, it would then be a simple matter of having a selectable terrain engine and then simply dropping in new components.
Hopefully this hasn't been rehashed to death too many times already, but I've tried to read up on all the terrain threads I could before posting.
Thoughts?
Dropping in a new terrain engine is, obviously, non-trivial, but I don't believe it is fundamentally insurmountable. From an engine standpoint (not tools) it looks like the requirements would be to abstract out the fundamental terrain representation and remove any existing implementation dependencies from the game body itself.
Ideally things like WaterBlock::mCameraSubmerged would instead become effects flags, thereby decoupling the game or client from knowing about WaterBlocks directly. Material properties would be queried indirectly instead of referenced directly from the TerrainBlock, etc.
Once that basic infrastructure was in place, it would then be a simple matter of having a selectable terrain engine and then simply dropping in new components.
Hopefully this hasn't been rehashed to death too many times already, but I've tried to read up on all the terrain threads I could before posting.
Thoughts?
#22
03/03/2003 (1:58 pm)
Yay, let me jump in here and agree with the need for a roadmap of where Torque is going from GG staff ... without that, we're a gaggle of geeks going every which way.
#23
03/03/2003 (2:18 pm)
Over on the TEP site folks were talking about a TEP fork. But those plans, and TEP itself sort of imploded. Baring a roadmap from GG with plans for extending TGE I'd be very interested in lending support to a 3rd party fork of TGE. There are numerous things that *should* be in the main tree but aren't it's a royal PITA to keep up to date even with vendor branches in CVS.
#24
Agreed. I don't mean use COM itself, I use the term "lite COM style" simply as an example of what I'm talking about. COM works this way on it's simplest level.
In Destiny3D, functionality is split into dlls. Each dll has one header file to the outside world. This header contains pure virtual interface classes and some factory function definitions. In the outside world, code defines a class pointer using the interface class and then calls the factory function to create the object and return the pointer to it. The only things the dll exports are the factory functions.
For singleton objects, the dll creates the object as soon as it's loaded and the factory function returns it's pointer to all callers.
This is probably pretty much what you are doing. It works real slick.
03/03/2003 (2:30 pm)
Brian,Agreed. I don't mean use COM itself, I use the term "lite COM style" simply as an example of what I'm talking about. COM works this way on it's simplest level.
In Destiny3D, functionality is split into dlls. Each dll has one header file to the outside world. This header contains pure virtual interface classes and some factory function definitions. In the outside world, code defines a class pointer using the interface class and then calls the factory function to create the object and return the pointer to it. The only things the dll exports are the factory functions.
For singleton objects, the dll creates the object as soon as it's loaded and the factory function returns it's pointer to all callers.
This is probably pretty much what you are doing. It works real slick.
#25
Mark: Yep. =)
03/03/2003 (2:35 pm)
JDS: The question is -- are we ALLOWED to fork the codebase on our own? I don't think so, because ideally it would be its own repository, and we'd have no way of controlling access to it. My guess is that the EULA allows us to do patches, but not to fork entire branches that are publicly accessible.Mark: Yep. =)
#26
Jinx again... CVS Repository thread... sitting, patiently awaiting any form of answer :)
-J
03/03/2003 (2:39 pm)
Brian,Jinx again... CVS Repository thread... sitting, patiently awaiting any form of answer :)
-J
#27
Just a quick note... one thing I have longed to do is replace Torque's interior/bsp code with QuakeII's preprocess(vis/light) and runtime system ... now that would be cool... though, of course GPL and the last I checked the QuakeII tools still ran $5k for non-GPL use...
Oh, don't know if you remember the infamous "Juggernaut for QuakeII"... but Canopy Games, a company I helped found in 1996, was behind that masterpiece.. 9 weeks from beginning to end... I still have the scars :)
-J
03/03/2003 (3:33 pm)
Brian,Just a quick note... one thing I have longed to do is replace Torque's interior/bsp code with QuakeII's preprocess(vis/light) and runtime system ... now that would be cool... though, of course GPL and the last I checked the QuakeII tools still ran $5k for non-GPL use...
Oh, don't know if you remember the infamous "Juggernaut for QuakeII"... but Canopy Games, a company I helped found in 1996, was behind that masterpiece.. 9 weeks from beginning to end... I still have the scars :)
-J
#28
Hey, very cool! I agree, the Q2 codebase is a hoot to mess with. I'm actually modding the hell out of it in parallel with the Torque engine. I'm doing something analogous to your PyTorque stuff -- I'm converting Q2 (with help) to using Lua for scripting instead of C DLLs.
How different are the Q2 and Torque interior engines?
03/03/2003 (3:38 pm)
Josh,Hey, very cool! I agree, the Q2 codebase is a hoot to mess with. I'm actually modding the hell out of it in parallel with the Torque engine. I'm doing something analogous to your PyTorque stuff -- I'm converting Q2 (with help) to using Lua for scripting instead of C DLLs.
How different are the Q2 and Torque interior engines?
#29
It would be very cool to be involved with a refactoring project.
Other than the terrain issues, ideas, there are a few other sticky issues in the engine, namely:
Hardcoded animation/movement in player.cc. Fps isn't just script, it's ground into the engine. This also includes the way the engine handles animations (Sequence0, Sequence1, etc..)
source files for the compiler, I don't see any lex or yacc definition files. A look at the bytecode interpreter for possible autobinding (if it doesn't already I've not even touched it yet).
The texture blender, the current one is asm, and it has a c++ vers, no one seems to know anything about it/maintain it, it's holding back texture size.
The file formats need to be documented, the exporter needs error handling/reporting. I've done some of this, the dump file at least tells you what part it's attempting instead of what succeeded.
As far as making a module plugin system. The 'com lite' is the way to go, this is something I've done many times before, it's the first step in any project I work on myself. My MMO engine (yes I made one, don't laugh, it's older than dirt), can take compiled C/C++(with the correct functions/'entry point'/registration), any COM object, or any .NET CLR compatible assembly, it even takes LPC. I'm not much for script integration as bison/yacc and flex/lex make me want to stick a pencil in my eyeball... mostly due to aforementioned LPC.
I'm wondering if we would be allowed to fork, but release patches only, and only to SDK holders?
regarding the q2 interior renderer, isn't that a bit dated? I'm asking seriously, not trying to be a smart ass.
regards
brad
03/03/2003 (4:34 pm)
I've only had the codebase a few weeks, looking back in the forums it seems that the last time the townsfolk got all riled up and made with the torches and pitchforks it resulted in the TEP, but all links leading there result in dead ends or to a forum with a grand total of 15 posts on it. Anyone know what's up with it?It would be very cool to be involved with a refactoring project.
Other than the terrain issues, ideas, there are a few other sticky issues in the engine, namely:
Hardcoded animation/movement in player.cc. Fps isn't just script, it's ground into the engine. This also includes the way the engine handles animations (Sequence0, Sequence1, etc..)
source files for the compiler, I don't see any lex or yacc definition files. A look at the bytecode interpreter for possible autobinding (if it doesn't already I've not even touched it yet).
The texture blender, the current one is asm, and it has a c++ vers, no one seems to know anything about it/maintain it, it's holding back texture size.
The file formats need to be documented, the exporter needs error handling/reporting. I've done some of this, the dump file at least tells you what part it's attempting instead of what succeeded.
As far as making a module plugin system. The 'com lite' is the way to go, this is something I've done many times before, it's the first step in any project I work on myself. My MMO engine (yes I made one, don't laugh, it's older than dirt), can take compiled C/C++(with the correct functions/'entry point'/registration), any COM object, or any .NET CLR compatible assembly, it even takes LPC. I'm not much for script integration as bison/yacc and flex/lex make me want to stick a pencil in my eyeball... mostly due to aforementioned LPC.
I'm wondering if we would be allowed to fork, but release patches only, and only to SDK holders?
regarding the q2 interior renderer, isn't that a bit dated? I'm asking seriously, not trying to be a smart ass.
regards
brad
#30
As an example, I spent the weekend cleaning up the Quake2 source base from the icculus.org fork and my diffs are over 650K in size as reported by cvs. That's a bit of a patch =)
From MarkF's comments, it's beginning to look like it'll be months before a complete refactor is done, so I'm of the mind that I'm just going to for the code base with what I have and hope for the best. I'm not that thrilled about it, since there are huge swaths of code that I have no experience or familiarity with, but on the plus side, I don't have to worry about having someone approve my patches, breaking my changes, and conversely, inadvertently breaking everyone else's patches and changes.
The Q2 rendering system is fairly dated, yes, but it's still quite serviceable depending on what you're trying to accomplish. It's still a lightmapped based BSP system with PVS and area portals, and the character system is laughable, but depending on your needs/goals, it's still good enough for most things.
03/03/2003 (6:14 pm)
I don't think there's any limit against forking, it's making the new fork available that's problematic. If you do it as patches, your fine, since you're not releasing a working engine -- that's what the resources here seem to be. But if you do a serious fork, there's a good chance that your diffs will be so huge that it's meaningless.As an example, I spent the weekend cleaning up the Quake2 source base from the icculus.org fork and my diffs are over 650K in size as reported by cvs. That's a bit of a patch =)
From MarkF's comments, it's beginning to look like it'll be months before a complete refactor is done, so I'm of the mind that I'm just going to for the code base with what I have and hope for the best. I'm not that thrilled about it, since there are huge swaths of code that I have no experience or familiarity with, but on the plus side, I don't have to worry about having someone approve my patches, breaking my changes, and conversely, inadvertently breaking everyone else's patches and changes.
The Q2 rendering system is fairly dated, yes, but it's still quite serviceable depending on what you're trying to accomplish. It's still a lightmapped based BSP system with PVS and area portals, and the character system is laughable, but depending on your needs/goals, it's still good enough for most things.
#31
The big things Q2 does over Q1 and Torque are:
1: Brush based BSP (MUCH less flakey than surface based)
2: Radiosity
3: Extremely tight PVS (over torque)
It wouldn't be trivial to remove the Torque interior code... but hey, dig around in map2dif/the rendering stuff and see if *you* would want to maintain it :)
Torque has LOD for it's interiors and obviously doesn't have the outside filling code/leaky map stuff... this stuff wouldn't be terribly hard to code in though, and in many cases you would WANT all Q2's great/efficient exterior stripping code(for instance, if you can't see any of the exterior brushes)... and I would take Quake II's areaportals and PVS over brush based portals anyday...
QuakeII tools aren't GPL and are quite expensive... so I never pursued this past sizing it up...
-J
03/03/2003 (6:38 pm)
Q2 has very solid BSP preprocess(in fact, Q3 is really just incremental)... the rendering engine is perhaps a bit crusty... but I always saw the real worth and genius of Quake technology being the preprocess tools... I mean look at all the Quake I-III level viewers :)The big things Q2 does over Q1 and Torque are:
1: Brush based BSP (MUCH less flakey than surface based)
2: Radiosity
3: Extremely tight PVS (over torque)
It wouldn't be trivial to remove the Torque interior code... but hey, dig around in map2dif/the rendering stuff and see if *you* would want to maintain it :)
Torque has LOD for it's interiors and obviously doesn't have the outside filling code/leaky map stuff... this stuff wouldn't be terribly hard to code in though, and in many cases you would WANT all Q2's great/efficient exterior stripping code(for instance, if you can't see any of the exterior brushes)... and I would take Quake II's areaportals and PVS over brush based portals anyday...
QuakeII tools aren't GPL and are quite expensive... so I never pursued this past sizing it up...
-J
#32
I am in total agreement with your two-phase refactoring of the terrain system. If you obtain a copy of TerrainManager, you'll find phase-one to be complete.
I did a lot of multiple-terrain modifications to the engine, which did entail a lot of overhead in the TerrainManager implementation and patching (specifically saving missions and terrain lighting). Still, refactoring the terrain engine would require doing this anyway, and it's already done.
Also, for any large-scale engine, you'll want to cut the terrain into blocks. The rest of the engine is not built to support this feature (collision/lighting). With TerrainManager, the engine has been updated to allow for multiple terrain blocks (with arbitrary implementations).
I'll send you the (old) TerrainManager header file to tide you over until J. Donavan Stanley gets a chance to finish his merge. If you're burning to work on something, spend a few days coding up the Chunked LOD engine. At best, you'll get to plug it into TerrainManager by the end of the week, and at worst you'll have to get your feet muddy to plug it into Torque from scratch.
--Bryan
bryan.turner@pobox.com
03/03/2003 (7:16 pm)
Brian,I am in total agreement with your two-phase refactoring of the terrain system. If you obtain a copy of TerrainManager, you'll find phase-one to be complete.
I did a lot of multiple-terrain modifications to the engine, which did entail a lot of overhead in the TerrainManager implementation and patching (specifically saving missions and terrain lighting). Still, refactoring the terrain engine would require doing this anyway, and it's already done.
Also, for any large-scale engine, you'll want to cut the terrain into blocks. The rest of the engine is not built to support this feature (collision/lighting). With TerrainManager, the engine has been updated to allow for multiple terrain blocks (with arbitrary implementations).
I'll send you the (old) TerrainManager header file to tide you over until J. Donavan Stanley gets a chance to finish his merge. If you're burning to work on something, spend a few days coding up the Chunked LOD engine. At best, you'll get to plug it into TerrainManager by the end of the week, and at worst you'll have to get your feet muddy to plug it into Torque from scratch.
--Bryan
bryan.turner@pobox.com
#33
Thanks for the update and offer. I might go ahead and do the chunked LOD system independently so that it can be plugged into either Torque or a Q2 engine, but at this point the feeling I'm getting is that it will be months, at the earliest, before GG will have a revamped revision of Torque. My instinct is that any changes done until then will be largely wasted since it doesn't seem like there'll be another major HEAD revision before then (given the history of updates to the cvs repository).
In other words, anything I (or you or others) do now that are significant architectural changes are, effectively, wasted work when thinking about the future engine.
So what I'm going to do, I think, is just rewrite the whole damn engine =)
This will allow me to remove large swaths of code I don't think I'll need, and it will also get my feet wet (or muddy, or stuck) since I'll be touching all the old code. When I'm done with the rewrite, I'll just send it back to GG and see what they think. I may even make an obligatory patch, but I highly doubt it would be relevant since I'll likely be removing entire files and switching to an entirely modular system.
I'm doing something identical right now on the Quake 2 code base, in fact.
03/03/2003 (7:35 pm)
Bryan,Thanks for the update and offer. I might go ahead and do the chunked LOD system independently so that it can be plugged into either Torque or a Q2 engine, but at this point the feeling I'm getting is that it will be months, at the earliest, before GG will have a revamped revision of Torque. My instinct is that any changes done until then will be largely wasted since it doesn't seem like there'll be another major HEAD revision before then (given the history of updates to the cvs repository).
In other words, anything I (or you or others) do now that are significant architectural changes are, effectively, wasted work when thinking about the future engine.
So what I'm going to do, I think, is just rewrite the whole damn engine =)
This will allow me to remove large swaths of code I don't think I'll need, and it will also get my feet wet (or muddy, or stuck) since I'll be touching all the old code. When I'm done with the rewrite, I'll just send it back to GG and see what they think. I may even make an obligatory patch, but I highly doubt it would be relevant since I'll likely be removing entire files and switching to an entirely modular system.
I'm doing something identical right now on the Quake 2 code base, in fact.
#34
I do like the idea of breaking the Terrain Manager manager out and making it easy to add a terrain implementation that fits the game you are designing. We would definitely incorporate this into the HEAD.
We still have many fixes/updates to the Torque to roll back in from Marble Blast after which we were planning on updating the label to 1.2.0.
As MarkF mentioned we will be refocusing on the Torque in the immediately after GDC. Our apologies for leaving the Torque on auto-pilot, but launching the publish arm of GarageGames has consumed every bit of our energies for the past several months.
03/04/2003 (9:34 am)
Hey guys, my apologies for seeing this thread earlier as some misinformation/assumptions about GG did creep in. It only took reading over 200 posts to catch up with you all, whew.I do like the idea of breaking the Terrain Manager manager out and making it easy to add a terrain implementation that fits the game you are designing. We would definitely incorporate this into the HEAD.
We still have many fixes/updates to the Torque to roll back in from Marble Blast after which we were planning on updating the label to 1.2.0.
As MarkF mentioned we will be refocusing on the Torque in the immediately after GDC. Our apologies for leaving the Torque on auto-pilot, but launching the publish arm of GarageGames has consumed every bit of our energies for the past several months.
#36
03/04/2003 (2:49 pm)
hey now that's a beautiful thing 
Torque Owner Prairie Games
Prairie Games, Inc.
I started a thread on the need to fork here. To me this is painfully obvious... keep in mind, this is the Tribes2 engine and was coded with specific requirements and deadlines... there is nothing *wrong* with it as such... but my, it could be better :)
I am doing a lot of work with Torque... and can contribute a fair amount... I am spending the day looking at the possibility of leveraging The Nebula Device's kernel/package system with Torque... no comments yet...
Regardless, I am about 2 steps from my own fork... it would be good for forkers to perhaps fork together? Any fork I work on will have kick ass Python support btw :)
-J