Pluggable terrain engine
by Brian Hook · in Torque Game Engine · 02/28/2003 (9:03 pm) · 36 replies
The existing terrain engine has some well documented limitations that I know others are working around, however the technology is decidedly "old-school". Newer terrain rendering technology exists (e.g. Ulrich's "Chunked LOD") that is, by and large, better for most modern computers.
Dropping in a new terrain engine is, obviously, non-trivial, but I don't believe it is fundamentally insurmountable. From an engine standpoint (not tools) it looks like the requirements would be to abstract out the fundamental terrain representation and remove any existing implementation dependencies from the game body itself.
Ideally things like WaterBlock::mCameraSubmerged would instead become effects flags, thereby decoupling the game or client from knowing about WaterBlocks directly. Material properties would be queried indirectly instead of referenced directly from the TerrainBlock, etc.
Once that basic infrastructure was in place, it would then be a simple matter of having a selectable terrain engine and then simply dropping in new components.
Hopefully this hasn't been rehashed to death too many times already, but I've tried to read up on all the terrain threads I could before posting.
Thoughts?
Dropping in a new terrain engine is, obviously, non-trivial, but I don't believe it is fundamentally insurmountable. From an engine standpoint (not tools) it looks like the requirements would be to abstract out the fundamental terrain representation and remove any existing implementation dependencies from the game body itself.
Ideally things like WaterBlock::mCameraSubmerged would instead become effects flags, thereby decoupling the game or client from knowing about WaterBlocks directly. Material properties would be queried indirectly instead of referenced directly from the TerrainBlock, etc.
Once that basic infrastructure was in place, it would then be a simple matter of having a selectable terrain engine and then simply dropping in new components.
Hopefully this hasn't been rehashed to death too many times already, but I've tried to read up on all the terrain threads I could before posting.
Thoughts?
#2
03/01/2003 (8:43 pm)
Are you planning on allowing more then one terrain block so that we can have bigger maps that aren't the same terrain repeated over and over?
#3
Anyway, Mark F is probably they guy you want to talk to if you want to do a terrain related head merge.
03/01/2003 (8:53 pm)
Pluggable terrain would be cool. I personally would like a terrain engine that allows "overhangs"...so that you could form caves out of terrain, have planetoid bodies, etc. A few weeks ago I dug into terrData.cc, thinking of modifying it to do this, but I think the changes would require a new way of representing the terrain instead of heightmaps. Thus probably a whole new terrain engine.Anyway, Mark F is probably they guy you want to talk to if you want to do a terrain related head merge.
#4
The goal here is to be able to have:
- no terrain engine
- existing terrain engine
- new terrain engine
The TerrainManager that others are working on will/does provide the ability to have multiple active terrain blocks, but I'm looking at something far more ambitious -- the ability to just plug in my own terrain engine based on the chunk LOD stuff that Thatcher Ulrich did (Google for that if you want information on it).
The first step is removing the dependency between TerrainBlock and everything else, which I've now done mostly.
03/01/2003 (8:54 pm)
I plan on having both more and less than that =)The goal here is to be able to have:
- no terrain engine
- existing terrain engine
- new terrain engine
The TerrainManager that others are working on will/does provide the ability to have multiple active terrain blocks, but I'm looking at something far more ambitious -- the ability to just plug in my own terrain engine based on the chunk LOD stuff that Thatcher Ulrich did (Google for that if you want information on it).
The first step is removing the dependency between TerrainBlock and everything else, which I've now done mostly.
#5
Heightmaps are fundamentally incapable of representing concavities in the terrain. What you'll want is a general mesh renderer for terrain, which should also be fairly easy to do. The key is having the infrastructure in place so that you can mix and match. My goal is such that if you load a .ter file, you get TerrainBlock stuff. If you load a .chu file, you get chunked LOD. And, if you wanted, I suppose there's no reason not to do something that's a general mesh terrain renderer for exactly what you're talking about.
In fact, something like that could conceivably be done very quickly if you were willing to limit the scope, i.e. a single terrain mesh, no level-of-detail. You could probably convert/extend modify the existing mesh rendering code to support that.
In order to interface with the rest of the engine (not including tools, which are pretty much specific to TerrainBlock) with my new stuff you basically only have to provide the following interfaces:
virtual bool sampleLighting( const Point3F &kPt, ColorI *pColor ) = 0;
virtual S32 getMPMIndex( const int kOff ) = 0;
virtual bool getTerrainHeight( const Point2F & pos, F32 * height ) = 0;
virtual U16 getTerrainHeight(U32 x, U32 y) = 0 ;
virtual float getHeightExtents( F32 *pfMinHeight, F32 *pfMaxHeight ) = 0;
sampleLighting is used by the scene lighting to determine ambient lighting conditions. getMPMIndex is a general material interface. getTerrainHeight() is used by the collision system and by the fluid/waterblock stuff. getHeightExtents() is used for the fog volume stuff.
03/01/2003 (8:59 pm)
John,Heightmaps are fundamentally incapable of representing concavities in the terrain. What you'll want is a general mesh renderer for terrain, which should also be fairly easy to do. The key is having the infrastructure in place so that you can mix and match. My goal is such that if you load a .ter file, you get TerrainBlock stuff. If you load a .chu file, you get chunked LOD. And, if you wanted, I suppose there's no reason not to do something that's a general mesh terrain renderer for exactly what you're talking about.
In fact, something like that could conceivably be done very quickly if you were willing to limit the scope, i.e. a single terrain mesh, no level-of-detail. You could probably convert/extend modify the existing mesh rendering code to support that.
In order to interface with the rest of the engine (not including tools, which are pretty much specific to TerrainBlock) with my new stuff you basically only have to provide the following interfaces:
virtual bool sampleLighting( const Point3F &kPt, ColorI *pColor ) = 0;
virtual S32 getMPMIndex( const int kOff ) = 0;
virtual bool getTerrainHeight( const Point2F & pos, F32 * height ) = 0;
virtual U16 getTerrainHeight(U32 x, U32 y) = 0 ;
virtual float getHeightExtents( F32 *pfMinHeight, F32 *pfMaxHeight ) = 0;
sampleLighting is used by the scene lighting to determine ambient lighting conditions. getMPMIndex is a general material interface. getTerrainHeight() is used by the collision system and by the fluid/waterblock stuff. getHeightExtents() is used for the fog volume stuff.
#6
It's possible to do volumetric type terrain but there is a patent on one of the best ways to do it (marching cubes). This fellow has done some good work and has links to others, he claims not to use marching cubes. Data storage is a problem but I remember seeing a 3d form of a bitmap somewhere... I'll dig
http://www.home.zonnet.nl/petervenis/index.htm
Brian,
Cool =D
03/01/2003 (9:00 pm)
JohnQ,It's possible to do volumetric type terrain but there is a patent on one of the best ways to do it (marching cubes). This fellow has done some good work and has links to others, he claims not to use marching cubes. Data storage is a problem but I remember seeing a 3d form of a bitmap somewhere... I'll dig
http://www.home.zonnet.nl/petervenis/index.htm
Brian,
Cool =D
#7
Have you given any thought to your work integrating with Torque's realtime editing?
-J
03/01/2003 (9:02 pm)
Brian,Have you given any thought to your work integrating with Torque's realtime editing?
-J
#8
Volumetric terrain is, er, overkill. It's possible, but as you say, the storage requirements are immense. You'd like store everything as a form of sparse oct-tree.
But frankly, it would be much more efficient and just as good just to use general meshes, possibly constructed from volume data.
Joshua,
I don't have plans to integrate my own stuff with the in-game tools, primarily because the chunk LOD algorithm requires a fairly length precomputation stuff and relies on using .BT files (although writing a converter to important .DEM or something similar wouldn't be difficult by any means). The in-game tools are pretty much hardcoded to the TerrainBlock system.
03/01/2003 (9:13 pm)
Brad,Volumetric terrain is, er, overkill. It's possible, but as you say, the storage requirements are immense. You'd like store everything as a form of sparse oct-tree.
But frankly, it would be much more efficient and just as good just to use general meshes, possibly constructed from volume data.
Joshua,
I don't have plans to integrate my own stuff with the in-game tools, primarily because the chunk LOD algorithm requires a fairly length precomputation stuff and relies on using .BT files (although writing a converter to important .DEM or something similar wouldn't be difficult by any means). The in-game tools are pretty much hardcoded to the TerrainBlock system.
#9
Frankly I've never seen an LOD behind simple stuff like what eberly lays out in his book for terrain that made much difference. It always seemed like the on the fly optimization evened out the gain in speed. Chunk LOD is the first one I've seen that looks nice and makes sense outside of theory land.
-brad
03/01/2003 (9:16 pm)
oh I'd never think of actually using it, I only mention it when people talk about overhangs, the collision would be expensive I'd think, hard to create, lack of tools, etc. An then yeah, then why not just use a big mesh.Frankly I've never seen an LOD behind simple stuff like what eberly lays out in his book for terrain that made much difference. It always seemed like the on the fly optimization evened out the gain in speed. Chunk LOD is the first one I've seen that looks nice and makes sense outside of theory land.
-brad
#10
-Jeff
03/02/2003 (1:14 pm)
You might also want to look into ROAM or ROAM 2.0 algorithms. They appear to be a bit more flexible than CLOD yet as fast and low on storage requirements.-Jeff
#11
03/02/2003 (2:16 pm)
ROAM algorithms are a form of CLOD, with the associated problems that CLOD algorithms entail. They can be modified to allow for deformable terrain (cf. Seamus McNally's Treadmarks stuff, I believe he did a write up on the ROAM modifications he performed), but they're still a net loss (IMO) vs. chunked LOD.
#12
I posted a note on the TerrainManager thread regarding your project. Looks like you've already gotten your feet muddy. :) I hope this message will save you from the dirty work!
I'll paraphrase from that post: TerrainManager does what you want, and it is already integrated into the editor, collision, lighting, etc. I believe your time would best be spent implementing Chunked LOD and attaching it to the TerrainManager interfaces rather than re-inventing the TerrainManager. Since the dirty work was already done last year, you should be able to avoid 99% of it now.
Please take a look!
--Bryan
bryan.turner@pobox.com
http://www.garagegames.com/index.php?sec=mg&mod=forums&page=result.thread&qt=4474
03/03/2003 (10:45 am)
Brian,I posted a note on the TerrainManager thread regarding your project. Looks like you've already gotten your feet muddy. :) I hope this message will save you from the dirty work!
I'll paraphrase from that post: TerrainManager does what you want, and it is already integrated into the editor, collision, lighting, etc. I believe your time would best be spent implementing Chunked LOD and attaching it to the TerrainManager interfaces rather than re-inventing the TerrainManager. Since the dirty work was already done last year, you should be able to avoid 99% of it now.
Please take a look!
--Bryan
bryan.turner@pobox.com
http://www.garagegames.com/index.php?sec=mg&mod=forums&page=result.thread&qt=4474
#13
It only took me one evening to do the basic refactoring, and I have a patch sitting on my hard drive ready to go. I have no idea though what to do with it at this point.
The stuff I've done so far does NOT include any of the MxN stuff and basically does the first pass abstraction. As I've detailed elsewhere, the important thing was using generalized interfaces for things like sampling the terrain lighting and height, which it sounds like TM also did.
Not to overemphasize this, but at this point I'd prefer not to fork off the main Torque engine too far, which is why I'd like to see my modifications (or something similar) pulled into the next revision so that future changes can be done non-destructively.
If anyone would like to see the patches, let me know and I'll figure out where to post them.
03/03/2003 (10:59 am)
The problem with TerrainManager (at least from the descriptions I've seen) are that it's not eligible for integration with HEAD anytime soon -- I swore I saw a comment somewhere that basically said that.It only took me one evening to do the basic refactoring, and I have a patch sitting on my hard drive ready to go. I have no idea though what to do with it at this point.
The stuff I've done so far does NOT include any of the MxN stuff and basically does the first pass abstraction. As I've detailed elsewhere, the important thing was using generalized interfaces for things like sampling the terrain lighting and height, which it sounds like TM also did.
Not to overemphasize this, but at this point I'd prefer not to fork off the main Torque engine too far, which is why I'd like to see my modifications (or something similar) pulled into the next revision so that future changes can be done non-destructively.
If anyone would like to see the patches, let me know and I'll figure out where to post them.
#14
@BH: I was the source of that comment. You might have misunderstood me. It's not that you CAN'T merge it with the GG head it that GG isn't likely to include it. They're also not likely to include ANY sweeping changes to the terrain system unless they start a fork to 2.0. Right now, it's my understanding that the only things going into head are the types of things you would typicily find in a dot release not the massive sweeping changes one might find in a major release.
Another very good reason TM isn't in HEAD is that it's never been presented to GG in the proper manner for it to be considered for inclusion (i.e. in patch form), that step at least will be done soon.
There are literaly TONS of patches for TGE that have not been, not likely ever will be, included as part of the stock TGE tree.
03/03/2003 (11:15 am)
"The problem with TerrainManager (at least from the descriptions I've seen) are that it's not eligible for integration with HEAD anytime soon -- I swore I saw a comment somewhere that basically said that."@BH: I was the source of that comment. You might have misunderstood me. It's not that you CAN'T merge it with the GG head it that GG isn't likely to include it. They're also not likely to include ANY sweeping changes to the terrain system unless they start a fork to 2.0. Right now, it's my understanding that the only things going into head are the types of things you would typicily find in a dot release not the massive sweeping changes one might find in a major release.
Another very good reason TM isn't in HEAD is that it's never been presented to GG in the proper manner for it to be considered for inclusion (i.e. in patch form), that step at least will be done soon.
There are literaly TONS of patches for TGE that have not been, not likely ever will be, included as part of the stock TGE tree.
#15
I think one different between TM and what I'm doing is the scope. I have made no attempt to actually replace the terrain engine, I've only changed interfaces and ensured that everything continues to work.
It's really a two-phase process, which I'm hoping would be more acceptable to the community at large. The first phase is to put in a new interface so that any future changes can be made nondestructively. This, to me, seems like a fundamentally good idea, since new subsystems of any type would have minimal impact on each other ideally (assuming everything operates via sub-classing). As part of the first phase, the existing implementations are unaltered by and large, and current missions and editors continue to work normally.
The second phase would be an incremental update to the system so that you can put in new terrain systems side-by-side. This would registering new terrain file types, etc.
The important thing is that each incremental update would retain backwards compatibility without forcing revisions of the mission and other data files.
That said, I haven't actually seen the TerrainManager stuff, so maybe I should before continuing my blathering =) Link?
03/03/2003 (11:26 am)
Ack, this worries me a bit.I think one different between TM and what I'm doing is the scope. I have made no attempt to actually replace the terrain engine, I've only changed interfaces and ensured that everything continues to work.
It's really a two-phase process, which I'm hoping would be more acceptable to the community at large. The first phase is to put in a new interface so that any future changes can be made nondestructively. This, to me, seems like a fundamentally good idea, since new subsystems of any type would have minimal impact on each other ideally (assuming everything operates via sub-classing). As part of the first phase, the existing implementations are unaltered by and large, and current missions and editors continue to work normally.
The second phase would be an incremental update to the system so that you can put in new terrain systems side-by-side. This would registering new terrain file types, etc.
The important thing is that each incremental update would retain backwards compatibility without forcing revisions of the mission and other data files.
That said, I haven't actually seen the TerrainManager stuff, so maybe I should before continuing my blathering =) Link?
#16
TM already has done much of what you're trying to accomplish (i.e. refactoring the interface to the Terrain engine so it's not so tightly coupled). I also put code in to maintain backwards compatibility (which I broke again last release).
Without a clear roadmap as to where GG is heading with the engine and without any clear idea of what they will or will not accept as part of HEAD I can't say for certain what will and will not get in. All I can say is that unless it comes from an employee, or associate chances are it's not getting in unless it's a bug fix.
We REALLY need some input from the GG folks.
03/03/2003 (11:33 am)
The Link is dead ATM as I was asked to pull it down until such a time as I have it packaged as a patch. That'll be ready no later than the end of the week. (Really I expect it tomorrow but shit happens so I'll pad it a bit)TM already has done much of what you're trying to accomplish (i.e. refactoring the interface to the Terrain engine so it's not so tightly coupled). I also put code in to maintain backwards compatibility (which I broke again last release).
Without a clear roadmap as to where GG is heading with the engine and without any clear idea of what they will or will not accept as part of HEAD I can't say for certain what will and will not get in. All I can say is that unless it comes from an employee, or associate chances are it's not getting in unless it's a bug fix.
We REALLY need some input from the GG folks.
#17
03/03/2003 (11:39 am)
Well that's a bummer. I've mailed them, I'll see what they have to say. If necessary I'll take 1.1.2 and just fork it off, or switch to the Quake2 engine =|
#18
I've worked quite a bit with Surreal Software's Riot engine (they used it for the PC game Drakan, way back) that's capable of doing the things you describe *with* a heightmap engine. It basically layers heightmaps on top of heightmaps, with some of them marked as "ceilings" rather than "floors." As a way of building caves it's a major pain in the ass :), but it might be something like you're imagining.
Personally, I think you'd be better off figuring out a way of making arbitrary geometry that plugs into height maps. Take that as a perspective from one who builds the stuff (I'm not 100% clear as to what performance advantage each offers). Seems like more could be done to make DTS geometry usable as terrain; the 8 collision-object-limit is particularly heinous.
Sorry to butt into what's obviously tech-heavy conversation with little tech info to offer... but it's a subject that's near and dear to me :)
-Spencer
03/03/2003 (11:46 am)
John: Quote:I personally would like a terrain engine that allows "overhangs"...so that you could form caves out of terrain,
I've worked quite a bit with Surreal Software's Riot engine (they used it for the PC game Drakan, way back) that's capable of doing the things you describe *with* a heightmap engine. It basically layers heightmaps on top of heightmaps, with some of them marked as "ceilings" rather than "floors." As a way of building caves it's a major pain in the ass :), but it might be something like you're imagining.
Personally, I think you'd be better off figuring out a way of making arbitrary geometry that plugs into height maps. Take that as a perspective from one who builds the stuff (I'm not 100% clear as to what performance advantage each offers). Seems like more could be done to make DTS geometry usable as terrain; the 8 collision-object-limit is particularly heinous.
Sorry to butt into what's obviously tech-heavy conversation with little tech info to offer... but it's a subject that's near and dear to me :)
-Spencer
#19
I'd be all for helping out any group that would be interesting in moving the engine code to this style of structure. It's just more work than I could tackle on my own.
03/03/2003 (1:30 pm)
Personally, I think the whole engine should be pulled apart and coded into plugable, replacable modules using something like a lite COM style (ie pure virtual interface classes fronting dll-based implementations). This is the way I designed the Destiny3D engine from the start. It creates an easily updatable, mix and match modular structure. Basically, you can plug any implementation in that adheres to the interface without breaking anything else. Using dynamic libs allows you to drop in a replacement dll without disturbing any other engine components or application code. Another thing this allows is numerous implementations to exist in CVS without forks of any kind.I'd be all for helping out any group that would be interesting in moving the engine code to this style of structure. It's just more work than I could tackle on my own.
#20
That's my assessment as well. I don't think you need to use COM with all the associated baggage, just using good old fashioned clean coding + strong dependency analysis is enough to get you there.
My own engine does this with video devices, audio devices, etc. I have a pure abstract interface class which is all the rest of the engine knows about, i.e. it gets a HVideoDevice*, and then inside the guts I choose whether to make a HGDIVideoDevice or a HGAPIVideoDevice or an HXF86VideoDevice, etc.
It would be REALLY nice to get some guidance from the GG folks right now.
03/03/2003 (1:32 pm)
Mark,That's my assessment as well. I don't think you need to use COM with all the associated baggage, just using good old fashioned clean coding + strong dependency analysis is enough to get you there.
My own engine does this with video devices, audio devices, etc. I have a pure abstract interface class which is all the rest of the engine knows about, i.e. it gets a HVideoDevice*, and then inside the guts I choose whether to make a HGDIVideoDevice or a HGAPIVideoDevice or an HXF86VideoDevice, etc.
It would be REALLY nice to get some guidance from the GG folks right now.
Torque Owner Brian Hook
The basic extent of the changes are:
- added a TerrainEngine class. This is the public interface for terrain operations. The hierarchy is:
SceneObject
|
TerrainEngine
|
TerrainBlock
TerrainEngine currently has only a very few pure virtual functions defined. More can be added as necessary. Currently they are getTerrainHeight, sampleLighting, getMPMIndex, and getTerrainExtents.
- added terrain/terrEngine.h
- modified existing files that unnecessarily included terrain/terrData.h to include terrEngine.h instead or, ideally, no terrain info. These files include:
engine/collision/convex.cc
engine/collision/gjk.cc
engine/game/game.cc
engine/game/main.cc
engine/game/player.cc
engine/game/showTSShape.cc
engine/game/fx/fxFoliageReplicator.cc
engine/game/fx/fxShapeReplicator.cc
engine/game/fx/lightning.cc
engine/game/fx/splash.cc
engine/game/vehicles/hoverVehicle.cc
engine/game/vehicles/vehicle.cc
engine/game/vehicles/wheeledVehicle.cc
engine/sceneGraph/sceneGraph.cc
- modified code that reads TerrainBlock information directly to use the new interface class. Specifically:
engine/editor/missionAreaEditor.cc
engine/editor/terraformer.cc
engine/editor/terrainEditor.cc
engine/game/game.cc
engine/game/player.cc
engine/game/showTSShape.cc
engine/game/fx/lightning.cc
engine/game/vehicles/hoverVehicle.cc
engine/game/vehicles/vehicle.cc
engine/game/vehicles/wheeledVehicle.cc
engine/sceneGraph/sceneGraph.cc
engine/sceneGraph/sceneLighting.cc
engine/sceneGraph/sceneTraversal.cc
engine/sceneGraph/sceneObject.cc
engine/terrain/fluid.h
engine/terrain/fluid.cc
engine/terrain/terrData.cc
engine/terrain/terrData.h
+engine/terrain/terrEngine.h
engine/terrain/terrRender.cc
engine/terrain/terrRender2.cc
engine/terrain/waterBlock.cc
The scene graph's current terrain is now a TerrainEngine* instead of a TerrainBlock*. I moved the overloaded terrain proxy operators into the .cc file in sceneLighting.h so that everything that needed scene lighting didn't also need to know about the internal details of TerrainBlock.
sceneObject no longer directly samples the lightmap data, instead it calls TerrainEngine->sampleLighting, which will allow new lighting mechanisms to be added later with different terrain engines.
The fluid class now uses a TerrainEngine instead of a hardcoded 256x256 heightmap.
The waterblock class now uses a TerrainEngine*.
No changes to functionality were made, so it operates just like it used to, this basically just puts the infrastructure in place for a pluggable terrain engine.
WaterBlock will need to be updated/modified at some point as well, however for now it works fine, it just assumes that the terrain is a TerrainBlock of specific dimensions (safe assumption for now).
I believe my changes have been pretty much safe and sound, however if someone wants to test them, just let me know and I'll figure out how to package the diffs. If these changes look stable, then I'll submit them for the next HEAD update.
Once they're in, the next thing to do will be to have a new set of terrain file types, and then it will know whether to make a TerrainBlock or something else at load time. This will also have the side effect of allowing no terrain cleanly (i.e. the first pass would be a DummyTerrainEngine class).
Thoughts?