Introducing: T3D Terrain Master!
by Chris Calef · 11/25/2012 (7:18 pm) · 44 comments
Believe it or not, I am actually writing a GG blog which is not about Ecstasy Motion!!!
The subject of this one is near and dear to my heart, however. Little Known Fact: In a past life, before I switched careers into game programming, I actually received a Bachelor's degree in Geography from the University of Oregon.
I have always liked maps, and have always wanted to take on the problem of importing real life terrain into Torque. I have played around with it multiple times, but have always been stopped by the sheer volume of the data, as well as the lack of an infinite terrain paging system in Torque.
A few weekends ago, I decided it was time to quit playing around and really dig in.
Above is the first video demonstrating the Terrain Master in action. I won't go into an incredible amount of detail here, because I just did so in this readme file... check it out if you want to see all the details.
In short, what this project offers is two distinct services: A) it imports very large terrain datasets, stored in raw binary matrix form as exported by the free program 3DEM, and saves them as a grid of .ter files, and B) it pages through these files as the player or free camera moves around the world.
The dataset I'm using here represents a squarish area about 50 km on a side, covering the southern end of Oregon's Willamette Valley. I created this dataset by downloading 24 DEM files from a website called GeoCommunity (geocomm.com). See the readme above for complete instructions, but to make a long story short, I loaded all 24 DEM files into 3DEM, exported the data to a binary floating point matrix file, and then accessed that file to make the set of .ter files.
From that point, the files still have no lightmaps, normals, textures, or anything else to make them interesting. This is where L3DT comes in. Many Torque users should be familiar with this app, it is a lifesaver - although one of my biggest problems at the moment is getting the time to figure out how to script it or write a plugin for it that can process all of these files as a batch... but I get ahead of myself.
Above is the final image of the entire region included by my Terrain Master test project. You can download these terrains from BAG at the following URL:
SouthernWillamette.zip
If you download the whole project and try to use it, (read the readme first!!!) you will find yourself starting in the lower left corner of that map.
http://github.com/ChrisCalef/T3DTerrainMaster
Good luck, and happy mapping!
(But here's one more video of it in action, just for good measure)
About the author
#22
@chris, please take a look at some of those links, especially the virtual terrain project site, as it's entire existence is about making a digital earth.
I'm very excited at where this could be going but like I said earlier it really depends on if people want to work on it or not, but I am definitely willing to throw whatever I can do artistically into the project (as well as some cash) to finally make this happen.
Thanks guys!
11/30/2012 (7:17 pm)
Man, I'm so glad someone else wants spherical terrain as badly as I do... I've been working with another member, but he's been busy so work on it has completely stalled. I'm an artist with no coding skills whatsoever, so it's frustrating trying to get something like this accomplished and each effort ends in a dead end, in fact the first thread I've ever made was about this subject, http://www.garagegames.com/community/forums/viewthread/126200. On it I have a bunch of links to OPEN SOURCE planetary sims that will help get this implemented. Bill Vee I think was the closest to have spherical terrain implemented for the torque game engine but that too is pretty much vaporware.@chris, please take a look at some of those links, especially the virtual terrain project site, as it's entire existence is about making a digital earth.
I'm very excited at where this could be going but like I said earlier it really depends on if people want to work on it or not, but I am definitely willing to throw whatever I can do artistically into the project (as well as some cash) to finally make this happen.
Thanks guys!
#23
Also the black edge is indeed something to do with the shadow map torque generates for the terrain, its not in the texture. I did intend to look at that some point myself as it bugs the hell out of me too...
12/01/2012 (8:06 am)
Hey Chris, when youre spawning the terrains, you could use the "screen error" setting for your LOD's, so when youre at a long distance from it have it set at 512 (or possibly 256, not sure off hand what the 'cap' is), then as you get closer you could scale the LOD's in with say 512/128/32.Also the black edge is indeed something to do with the shadow map torque generates for the terrain, its not in the texture. I did intend to look at that some point myself as it bugs the hell out of me too...
#24
I take it you must be making some sort of space sim, but for that you might look into how important it is for you to have absolutely seamless transition from space flight to ground FPS play... because obviously you would want far, far less resolution for the space flight, and even for the landing sequence, than you would want for an FPS part of your game.
This DEM data I'm working with is probably not the best source for a space flight view, as it only covers the US (as far as I know) and it's much higher resolution than you would want. If you have lower res data for the whole planet, though, I'd also like to see that, and who knows, might even take a stab at solving the spherical issues.
I don't see much point in trying to make spherical waterblocks and fix issues re: player interaction with spherical terrain though, unless you're literally making The Little Prince in 3D, otherwise it just wouldn't seem relevant. Why not just switch to "flat mode" when your player gets out of the spaceship?
12/01/2012 (9:56 am)
@Agent: Just out of curiosity, what kind of project are you trying to do? I wasn't actually going to go spherical for this one, since for any small patch of the planet's surface it works quite well to pretend it's flat. I take it you must be making some sort of space sim, but for that you might look into how important it is for you to have absolutely seamless transition from space flight to ground FPS play... because obviously you would want far, far less resolution for the space flight, and even for the landing sequence, than you would want for an FPS part of your game.
This DEM data I'm working with is probably not the best source for a space flight view, as it only covers the US (as far as I know) and it's much higher resolution than you would want. If you have lower res data for the whole planet, though, I'd also like to see that, and who knows, might even take a stab at solving the spherical issues.
I don't see much point in trying to make spherical waterblocks and fix issues re: player interaction with spherical terrain though, unless you're literally making The Little Prince in 3D, otherwise it just wouldn't seem relevant. Why not just switch to "flat mode" when your player gets out of the spaceship?
#25
For the height data the whole world has been digitized and turnef into 250 m/pixel data(links to data can be found on the virtual terrain project site I mentioned earlier. With that, I thought you might be able to use the same ROAM technique so that its not displaying the full resolution height data from a distance, just like in google earth.
I know that physics does change a bit because gravity is normally just neg y on a flat surface but on a planet its actually pointing towards the center and such, but I've seen bill vee do it with an earlier version also with water blocks and atmospheric scattering. I honestly thought what he was doing with that project was the single coolest change to torque, but that's me. Anyway i'm using your importer to do a demo of a game which is set in san fran, hopefully I'll have something to show soon. The only incentive I can offer you to work on spherical terrain would be a few hundred dollars (for work done and exclusivity) and all research I have done. So yeah hopefully we can make something happen?
12/01/2012 (3:09 pm)
The project I want to put together is a planet sim. What was started was a cube that was spherized, and I think thats mostly all that the member created in t3d. For the project all games start from space with the earth in view, its kinda like a select screen that lets you rotate around, select level or whatever, and then a google earth type zoom down to the surface, I felt that constant flat terrain wouldnt help that effect.For the height data the whole world has been digitized and turnef into 250 m/pixel data(links to data can be found on the virtual terrain project site I mentioned earlier. With that, I thought you might be able to use the same ROAM technique so that its not displaying the full resolution height data from a distance, just like in google earth.
I know that physics does change a bit because gravity is normally just neg y on a flat surface but on a planet its actually pointing towards the center and such, but I've seen bill vee do it with an earlier version also with water blocks and atmospheric scattering. I honestly thought what he was doing with that project was the single coolest change to torque, but that's me. Anyway i'm using your importer to do a demo of a game which is set in san fran, hopefully I'll have something to show soon. The only incentive I can offer you to work on spherical terrain would be a few hundred dollars (for work done and exclusivity) and all research I have done. So yeah hopefully we can make something happen?
#26
Ive been trying to accomplish this, Not sure If I want to go spherical world, but its crossed my mind in the past, as far as allowing players to take off, and colonize other worlds, and build on them.. I would currently be happy, and content with one full world as a sandbox..
Currently Ive been throwing around ideas, in theory in my mind, and looking @ the engine code on terrain..
One of My ideas was to take the original ground plane, You know the one you can walk for ever on that is not editable.. Well to place that under your world, or make it editable. Use the ground plane as a form of GPS on POS X/y coordinates, and to do checks between the ground plane. Your characters location which will be called when either a new tile pages the new terrain tile on top of that plane, while your previous POS data is destroyed as you leave the previous tile, that updates your FOV to POS X/Y positions while the map is loaded in Z in relation to the tile page system.. So, basically Your X/Y always remains as the source of origin 0/0, and the only thing that changes is the Z height.
This remind me of how Ultima Online use to kinda page their tiles to the player.. When you moved in the game the map moved not the character. Your character always stayed in the same location, just looked like an illusion of movement.. The Map was server side, while the need tiles were pages to the character, and buffered in, and out of memory as needed.. Even tho it was 2D, I dont see why the same concept cannot work in 3D as long as the Z location is kept accurate. This pretty much means a whole server can have the whole map on it thus freeing up the clients memory as a need to be basis, for other things..
12/02/2012 (7:18 am)
I dont know how to say this other then... Your work is amazingly AWESOME!!!!!!!!!! Ive been trying to accomplish this, Not sure If I want to go spherical world, but its crossed my mind in the past, as far as allowing players to take off, and colonize other worlds, and build on them.. I would currently be happy, and content with one full world as a sandbox..
Currently Ive been throwing around ideas, in theory in my mind, and looking @ the engine code on terrain..
One of My ideas was to take the original ground plane, You know the one you can walk for ever on that is not editable.. Well to place that under your world, or make it editable. Use the ground plane as a form of GPS on POS X/y coordinates, and to do checks between the ground plane. Your characters location which will be called when either a new tile pages the new terrain tile on top of that plane, while your previous POS data is destroyed as you leave the previous tile, that updates your FOV to POS X/Y positions while the map is loaded in Z in relation to the tile page system.. So, basically Your X/Y always remains as the source of origin 0/0, and the only thing that changes is the Z height.
This remind me of how Ultima Online use to kinda page their tiles to the player.. When you moved in the game the map moved not the character. Your character always stayed in the same location, just looked like an illusion of movement.. The Map was server side, while the need tiles were pages to the character, and buffered in, and out of memory as needed.. Even tho it was 2D, I dont see why the same concept cannot work in 3D as long as the Z location is kept accurate. This pretty much means a whole server can have the whole map on it thus freeing up the clients memory as a need to be basis, for other things..
#27
For the height data the whole world has been digitized and turnef into 250 m/pixel data(links to data can be found on the virtual terrain project site I mentioned earlier. With that, I thought you might be able to use the same ROAM technique so that its not displaying the full resolution height data from a distance, just like in google earth.
I know that physics does change a bit because gravity is normally just neg y on a flat surface but on a planet its actually pointing towards the center and such, but I've seen bill vee do it with an earlier version also with water blocks and atmospheric scattering. I honestly thought what he was doing with that project was the single coolest change to torque, but that's me. Anyway i'm using your importer to do a demo of a game which is set in san fran, hopefully I'll have something to show soon. The only incentive I can offer you to work on spherical terrain would be a few hundred dollars (for work done and exclusivity) and all research I have done. So yeah hopefully we can make something happen?
12/02/2012 (12:07 pm)
The project I want to put together is a planet sim. What was started was a cube that was spherized, and I think thats mostly all that the member created in t3d. For the project all games start from space with the earth in view, its kinda like a select screen that lets you rotate around, select level or whatever, and then a google earth type zoom down to the surface, I felt that constant flat terrain wouldnt help that effect.For the height data the whole world has been digitized and turnef into 250 m/pixel data(links to data can be found on the virtual terrain project site I mentioned earlier. With that, I thought you might be able to use the same ROAM technique so that its not displaying the full resolution height data from a distance, just like in google earth.
I know that physics does change a bit because gravity is normally just neg y on a flat surface but on a planet its actually pointing towards the center and such, but I've seen bill vee do it with an earlier version also with water blocks and atmospheric scattering. I honestly thought what he was doing with that project was the single coolest change to torque, but that's me. Anyway i'm using your importer to do a demo of a game which is set in san fran, hopefully I'll have something to show soon. The only incentive I can offer you to work on spherical terrain would be a few hundred dollars (for work done and exclusivity) and all research I have done. So yeah hopefully we can make something happen?
#28
most helpful terrain painting resource
to the project. If you're playing with TerrainMaster and pull changes you should be able to use it.
I added a button to call up autoLayers on the terrain painter tools palette, and also made the Undo logic functional or not depending on a bool in the call to autoMaterialLayer(), otherwise it's a direct copy of Konrad's work.
To use it, go to terrain painter, add some new layers with appropriate textures, and then select one of your layers and hit the new button on the tools palette. Then enter your max/min heights and slopes for applying this texture, and hit Generate.
Next step will be to set up these parameters in advance and auto-apply them for each terrain as I generate them... will let you know when that works.
12/02/2012 (3:36 pm)
So hey, exciting news on the terrain texturing front... after beating my head against the problem of trying to automate L3DT for a while, big thanks to Andy Wright (for pointing it out) :-P and Konrad Kiss (for writing it), but I finally became aware of and added Konrad's to the project. If you're playing with TerrainMaster and pull changes you should be able to use it.
I added a button to call up autoLayers on the terrain painter tools palette, and also made the Undo logic functional or not depending on a bool in the call to autoMaterialLayer(), otherwise it's a direct copy of Konrad's work.
To use it, go to terrain painter, add some new layers with appropriate textures, and then select one of your layers and hit the new button on the tools palette. Then enter your max/min heights and slopes for applying this texture, and hit Generate.
Next step will be to set up these parameters in advance and auto-apply them for each terrain as I generate them... will let you know when that works.
#30
12/02/2012 (5:53 pm)
god im gonna have to hook you up with some textures ;)
#31
Actually, if you have some you wouldn't mind committing to the project, and that are also legally free and clear for open sourcing, it would really help a _lot_ if you could send those ones specially marked.
Otherwise, yeah at least for demo work I'd love to have whatever you can give me, ESPECIALLY a northwest dark forest-covered-mountain texture or three. Haven't been able to find much yet in that regard.
Thanks again to both you and Konrad though(!!) - amazing the things like valuable resources that can slip right past when you're busy coding! :-P
12/03/2012 (10:03 am)
ROFL, Andy, yes please!!! :-)Actually, if you have some you wouldn't mind committing to the project, and that are also legally free and clear for open sourcing, it would really help a _lot_ if you could send those ones specially marked.
Otherwise, yeah at least for demo work I'd love to have whatever you can give me, ESPECIALLY a northwest dark forest-covered-mountain texture or three. Haven't been able to find much yet in that regard.
Thanks again to both you and Konrad though(!!) - amazing the things like valuable resources that can slip right past when you're busy coding! :-P
#32
Also gonna get the GIT stuff set up on here and port over and (possibly) finish off the ECK pack time permitting...
[Update] Hunch seems correct, its something to do with the terrain tile skirts... managed to exaggerate the black line shadowing, working on removing it now, fingers crossed anyway...
[Update Two] ok nearly fixed the black edges, tho somethings gone wonky with colission at the edge now instead, working on that ...
12/03/2012 (2:03 pm)
Ive got a hunch on the cause of the black lines, gonna look into it tonight. Just sorting out my laptop properly so I can compile on here then gonna jump back into the code and see if im on the right track...Also gonna get the GIT stuff set up on here and port over and (possibly) finish off the ECK pack time permitting...
[Update] Hunch seems correct, its something to do with the terrain tile skirts... managed to exaggerate the black line shadowing, working on removing it now, fingers crossed anyway...
[Update Two] ok nearly fixed the black edges, tho somethings gone wonky with colission at the edge now instead, working on that ...
#33
I keep getting "couldn't find terrain master, checking again!"
it worked perfectly the first time, I then upped the loadDistance and dropDistance, now it constantly gives that error, even after resetting the distances.
any idea why?
cheers.
EDIT: disregard, my mistake, works great
12/08/2012 (1:17 am)
I'm having an issue I cant work out,I keep getting "couldn't find terrain master, checking again!"
it worked perfectly the first time, I then upped the loadDistance and dropDistance, now it constantly gives that error, even after resetting the distances.
any idea why?
cheers.
EDIT: disregard, my mistake, works great
#34
12/08/2012 (1:18 am)
@Andy, any success with the edges?
#35
Had a nasty hack fix going, by not rendering the last pixel around the terrain lol, but colission remains
12/08/2012 (10:54 am)
nah hit a brick wall, also been a bit tied up working on somethin else.Had a nasty hack fix going, by not rendering the last pixel around the terrain lol, but colission remains
#36
If you have it commented out but you want terrain paging to start, you can just go "tryCheckTerrain();" once in the console and it will start the schedule running.
The next thing on my list, whenever I have time to get to it, is to automate Konrad's terrain painting system so it adds several materials, each with its own rules, to each terrain as it gets loaded... would gladly accept help if anyone wanted to make a fork and go to town on it...
Re: the edges, I'm also quite flummoxed, would love some insights on that from anyone who really understands Torque's lighting system. Andy did observe quite helpfully that it seems to be getting shadows from the other side of the terrain, ie when the terrain at one edge is higher than the corresponding terrain on the other side of the block, then the shadow is on the other side of the line, if lower then the shadow is on this side of the line. Still wasn't enough for either of us to figure it out yet, though.
12/09/2012 (2:48 pm)
@deepscratch: glad to hear it's working! One thing to know about that I may not have documented adequately is the tryCheckTerrain() function, which I turn on and off in player.cs (Armor::onAdd) depending on whether I am going to be importing new terrain into the system (comment it out) or using the terrain paging system normally (uncomment it). If you have it commented out but you want terrain paging to start, you can just go "tryCheckTerrain();" once in the console and it will start the schedule running.
The next thing on my list, whenever I have time to get to it, is to automate Konrad's terrain painting system so it adds several materials, each with its own rules, to each terrain as it gets loaded... would gladly accept help if anyone wanted to make a fork and go to town on it...
Re: the edges, I'm also quite flummoxed, would love some insights on that from anyone who really understands Torque's lighting system. Andy did observe quite helpfully that it seems to be getting shadows from the other side of the terrain, ie when the terrain at one edge is higher than the corresponding terrain on the other side of the block, then the shadow is on the other side of the line, if lower then the shadow is on this side of the line. Still wasn't enough for either of us to figure it out yet, though.
#37
12/09/2012 (2:55 pm)
Ive got a feeling theres something redundant in the code from when TGEA used to auto tile a terrain infinitley, it would auto tile the terrain for you and stitch it. this looks to be acting the same way, where your shadows are on the edge of the map, you cant run off the edge, as there is colission there where it is stitching up against the other height on the opposite side of the map. I think disabling that is the key... but im fucked if i can find it, also had no replies to a forum post about it :/
#38
Hmmm...
12/09/2012 (6:58 pm)
You're totally right about the collision thing Andy, I thought that might be because of the geometry change you were doing at the edges, but mine have the same behavior, I can't cross an edge on foot if that edge is lower than the corresponding edge on the opposite side of the tile.Hmmm...
#39
12/09/2012 (7:00 pm)
because it is 'drawing' a extra strip on the top and right edge of the terrain's rendered area. this invisible terrain is whats latching onto the height on the opposite side in what looks like an attempt to automatically tile the terrain. presumably redundant code from tgea unless at some point someone was working on getting terrain to tile and seam up automatically...
#40
12/18/2012 (2:06 pm)
Andy, I read somewhere that some user on Unity was having the same issue with getting the terrain to tile, and seam up right. I think their solution was editing the terrain manager to recognize the 4 terrain tiles, and connect the corners of the terrain adjusting the z height automatically leaving it seamless. While doing so, the rest of the terrain was paged in in a few tiles in advanced, and let go as needed. 
Associate Chris Calef
BrokeAss Games
In terms of Terrain Master then, a complicating factor is I'm thinking I'd like to have at least two LODs present, eventually, so that I could load a truly huge map (maybe 100 km square) at very low resolution, showing off the entire viewshed, and then cut chunks out of it local to the observer, to be replaced by higher resolution terrain tiles.
In that scheme I guess I'd just have to make up another distance factor for terrain, smaller than the visibility distance, and use that in the scoping decision.