limit on heightfield import size?
by Steve · in Torque 3D Professional · 05/09/2009 (1:18 pm) · 24 replies
Is there a limit on the size of heightfields which can be imported successfully? I am getting a "unhandled exception" error anytime I try to load anything over 4096 x 4096.
#2
05/09/2009 (6:29 pm)
Didn't the old atlas terrain system allow terrains bigger than 4096? Maybe I am doing something wrong, but it only took me 4 minutes to run the avatar from one end of the 4096 x 4096 terrain to the other, that doesn't seem big enough for a RPG game. Is there a way to have multiple terrain blocks or zones or something and if so, could anyone please point me to where i could how to implement them?
#3
Also keep in mind that the terrain heightmap isn't the only thing that controls/affects the size of the terrain you can have in a torque world. For example you can increase the squaresize to increase the terrain size. Then factor in that you can use the mega terrain system to blend multiple terrains together and you get much larger terrains.
One other thing to keep in mind for a RPG or MMO is you wouldn't realisticly want to make one single terrain for your whole world. Why? Well you only get so many digits of precision (8) before data gets dropped, so what this means is that the further you get from the world origin (0,0,0) the less accurate data is going to get. This can result in all sorts of fun things like models, particles, projectiles, etc not being displayed where they actually should be :( This can be less of a concern with double precision data, but its not going to be a silver bullet that solves your issue either.
Perhaps if you are lucky Josh and/or Lara from GG can post some information on how they did the world terrain in Minions of Mirth with just plain old TGE.
05/09/2009 (7:05 pm)
Yuck, Atlas. There isnt a single dammed positive thing to say about it. Also keep in mind that the terrain heightmap isn't the only thing that controls/affects the size of the terrain you can have in a torque world. For example you can increase the squaresize to increase the terrain size. Then factor in that you can use the mega terrain system to blend multiple terrains together and you get much larger terrains.
One other thing to keep in mind for a RPG or MMO is you wouldn't realisticly want to make one single terrain for your whole world. Why? Well you only get so many digits of precision (8) before data gets dropped, so what this means is that the further you get from the world origin (0,0,0) the less accurate data is going to get. This can result in all sorts of fun things like models, particles, projectiles, etc not being displayed where they actually should be :( This can be less of a concern with double precision data, but its not going to be a silver bullet that solves your issue either.
Perhaps if you are lucky Josh and/or Lara from GG can post some information on how they did the world terrain in Minions of Mirth with just plain old TGE.
#4
That said... even a 1 meter per pixel... a 4k by 4k terrain is fairly big. Move that to 4 meters per pixel and you got an 16k by 16k terrain... bigger than you would need in any game involving walking humans.
As far as using multiple TerrainBlocks in one map... you could do that, but we haven't really aimed at supporting it officially. So there are bound to be some issues.
05/09/2009 (9:16 pm)
As Logan was saying... you do have a +/- 10,000 or so units from the origin before you start loosing precision. So a single terrain... at least without any special handling for recentering the simulation... isn't ideal.That said... even a 1 meter per pixel... a 4k by 4k terrain is fairly big. Move that to 4 meters per pixel and you got an 16k by 16k terrain... bigger than you would need in any game involving walking humans.
As far as using multiple TerrainBlocks in one map... you could do that, but we haven't really aimed at supporting it officially. So there are bound to be some issues.
#5
All you have to do is create new ones and/or import the heightmaps, once the terrains are generated you then move, alter, and paint as normal.
There is of course a resource hit for each successive terrain, but it is much, much less than the old Legacy terrain or the Mega-Terrain trick. Oh, and do keep in mind the precision issue that Logan mentioned.
05/09/2009 (10:20 pm)
Maybe it's not as expansive as Atlas terrains -- I loved those things! -- but the ease of in-game design and the usage of opacity layers, plus materials more than makes up for it. You can easily abuse the terrain system and place any number of terrain blocks -- I only tested up to 9 of them so there could be a limit. It's up to you though to match the seams and line the individual blocks up. All you have to do is create new ones and/or import the heightmaps, once the terrains are generated you then move, alter, and paint as normal.
There is of course a resource hit for each successive terrain, but it is much, much less than the old Legacy terrain or the Mega-Terrain trick. Oh, and do keep in mind the precision issue that Logan mentioned.
#6
05/09/2009 (10:23 pm)
Just to add that the only real issues I've noticed with multiple terrain blocks so far, is that if you rotate one of them sometimes collision(s) fail for that block -- but not always. 
#7
its been stated that the terrain has to exist fully in memory.
Does this mean all terrains or just the one(s) you can 'see', where i'm heading with this is... if you have multiple terrains do you have a pseudo paging version of the terrain where t3d uses only the visible 'cells'
05/10/2009 (2:58 am)
please excuse the newbie type question, but..its been stated that the terrain has to exist fully in memory.
Does this mean all terrains or just the one(s) you can 'see', where i'm heading with this is... if you have multiple terrains do you have a pseudo paging version of the terrain where t3d uses only the visible 'cells'
#8
The whole terrain goes to RAM as you load it
thats exactly why anything larger than 4096x4096 would be a bad idea as that already gets you up to 1GB of RAM usage with all the collision etc
05/10/2009 (6:38 am)
There is no paging for terrainThe whole terrain goes to RAM as you load it
thats exactly why anything larger than 4096x4096 would be a bad idea as that already gets you up to 1GB of RAM usage with all the collision etc
#9
For example when the player moves to the west the next cell(terrain block) is loaded while one which is being left behind in the east is unloaded? Perhaps done incrementally, or some other way, in the background so there isn't much of a slowdown as loading is taking place?
05/10/2009 (8:40 am)
But would it be possible to implement a system like Bloodknight brings up.For example when the player moves to the west the next cell(terrain block) is loaded while one which is being left behind in the east is unloaded? Perhaps done incrementally, or some other way, in the background so there isn't much of a slowdown as loading is taking place?
#10
Thats excatly what i need for my game, paging terrains.
it is possible or rather "should" be possible, but since my game uses iso/ortho camera i guess its alot easier to actually implement then for a 3D first/3rd person view game.
Our system we plan on implementing is like this (sorry for crappy quality, just a quick sample, and all "-" should be "+" my bad) where each plane would have a "parent" plane for seemless transitions (terrain & texture) and painting over ect ect.
Alot of work, but o well, its needed for our project, so theres no way around it.
05/10/2009 (8:58 am)
@GerryThats excatly what i need for my game, paging terrains.
it is possible or rather "should" be possible, but since my game uses iso/ortho camera i guess its alot easier to actually implement then for a 3D first/3rd person view game.
Our system we plan on implementing is like this (sorry for crappy quality, just a quick sample, and all "-" should be "+" my bad) where each plane would have a "parent" plane for seemless transitions (terrain & texture) and painting over ect ect.
Alot of work, but o well, its needed for our project, so theres no way around it.
#11
To just try and understand your diagram better... you would have all the important plot stuff within the lightblue areas of each terrain block and a player would only move into the lightbrown area if they wanted to go to another block and the loading of the next terrain would only happen when they entered that lightbrown area? Could you explain more what you mean by a "parent plane"... something shared among all terrain blocks that would be always loaded in memory?
I understand if you don't want to share too much of your design ideas, but it would also be cool if someone could develop a system like this to such an extent that it could be incorporated into the engine where everybody could make use of it.
05/10/2009 (9:12 am)
@Bo To just try and understand your diagram better... you would have all the important plot stuff within the lightblue areas of each terrain block and a player would only move into the lightbrown area if they wanted to go to another block and the loading of the next terrain would only happen when they entered that lightbrown area? Could you explain more what you mean by a "parent plane"... something shared among all terrain blocks that would be always loaded in memory?
I understand if you don't want to share too much of your design ideas, but it would also be cool if someone could develop a system like this to such an extent that it could be incorporated into the engine where everybody could make use of it.
#12
05/10/2009 (9:27 am)
@Bo: If you want your AI to cross zones through pathfinding, you will also need to implement paging nodes (or whatever your AI uses for pathfinding). Maybe a global paging solution should be designed instead of one that concentrates on terrain.
#13
The terrain is made up of 1x1 km blocks, when player moves in/out over the red box, it unloads other terrains in memory or loads new ones depending if you moving in or out of the center green box.
The blue box is a "center" limit box, wich if crossed into, loads 3 terrain blocks instead of one, so we still have a full visible map (as you are close to 3 terrain blocks edge's).
Parent plane, means each plane is parentet to intersecting plane, wich means if you paint over or adjust high of the plane's edge (lets say +10 in height) then both planes (current + parent) edge gets raised same amount (+ deeper inside plane depending on brush size).
What it kind of means is, what happens on one, happens on the parent aswell if it affect/moves over to the other.
Hard to explain, hope you understand what i mean now :P
(also note, we do small maps since we are making a iso/ortho projection camera map, think ala "diablo" camera style), wich gives us way less view distance required.
@Konrad
Yeah, the AI for our sake is server controlled, move from x,y to new x,y and we use our own coordinate system, left buttom is 0,0 +(plane ID x 1024).
Lets say plane (A) is 1024x1024
Then plane (B) is 1024x1024 aswell, that makes map size 2048x2048, wich means if x=2000, y=2000 is where the AI controlled item is (then its really B plane at 976x976).
Wow so hard to explain these things xD (im not uber pro at english).
PS. almost feel like we hijacked the guys thread, but i guess its just sharing a idea in the same field, so hope its okay :P
05/10/2009 (9:39 am)
@GerryThe terrain is made up of 1x1 km blocks, when player moves in/out over the red box, it unloads other terrains in memory or loads new ones depending if you moving in or out of the center green box.
The blue box is a "center" limit box, wich if crossed into, loads 3 terrain blocks instead of one, so we still have a full visible map (as you are close to 3 terrain blocks edge's).
Parent plane, means each plane is parentet to intersecting plane, wich means if you paint over or adjust high of the plane's edge (lets say +10 in height) then both planes (current + parent) edge gets raised same amount (+ deeper inside plane depending on brush size).
What it kind of means is, what happens on one, happens on the parent aswell if it affect/moves over to the other.
Hard to explain, hope you understand what i mean now :P
(also note, we do small maps since we are making a iso/ortho projection camera map, think ala "diablo" camera style), wich gives us way less view distance required.
@Konrad
Yeah, the AI for our sake is server controlled, move from x,y to new x,y and we use our own coordinate system, left buttom is 0,0 +(plane ID x 1024).
Lets say plane (A) is 1024x1024
Then plane (B) is 1024x1024 aswell, that makes map size 2048x2048, wich means if x=2000, y=2000 is where the AI controlled item is (then its really B plane at 976x976).
Wow so hard to explain these things xD (im not uber pro at english).
PS. almost feel like we hijacked the guys thread, but i guess its just sharing a idea in the same field, so hope its okay :P
#14
I think I understand... its like trying to explain how to tie ones shoes isn't it. Easier to show then to tell. It will perhaps become clearer when I try doing something like this myself. Your diagram is pretty much the kind of thing I was thinking a system like this should be like. It shows I am on the right 'path' I suppose... ;)
Thanks for the explanations.
05/10/2009 (9:48 am)
@BoI think I understand... its like trying to explain how to tie ones shoes isn't it. Easier to show then to tell. It will perhaps become clearer when I try doing something like this myself. Your diagram is pretty much the kind of thing I was thinking a system like this should be like. It shows I am on the right 'path' I suppose... ;)
Thanks for the explanations.
#15
The average walking speed is 6-7 km/h. We'll pick the higher number of 7, and at 4096m, that should take you about 35 minutes to walk. Now, that's at a fairly quick pace, straight line, zero obstacles. Throw some stuff in the way, put some gear on the player to slow him down, and you're looking at well over an hour to reach the other side, if the player is still trying to rush.
Now, think of how many RPG's where it took you over an hour to reach the next load area. I imagine it's an extremely small list. Do you really think you need to be on that list yourself? I have a feeling you'll make do without it.
05/10/2009 (10:57 am)
Quote:Maybe I am doing something wrong, but it only took me 4 minutes to run the avatar from one end of the 4096 x 4096 terrain to the other, that doesn't seem big enough for a RPG game.
The average walking speed is 6-7 km/h. We'll pick the higher number of 7, and at 4096m, that should take you about 35 minutes to walk. Now, that's at a fairly quick pace, straight line, zero obstacles. Throw some stuff in the way, put some gear on the player to slow him down, and you're looking at well over an hour to reach the other side, if the player is still trying to rush.
Now, think of how many RPG's where it took you over an hour to reach the next load area. I imagine it's an extremely small list. Do you really think you need to be on that list yourself? I have a feeling you'll make do without it.
#16
and depending how detailed you want your terrain, a 4096 texture at the highest detail level doesnt really cover much ground and takes a huge level of resources especially if you want a good visibility distance and quality.
On the flip side of that if you dont require high levels of details you can make a 4096 terrain go a LONG way
05/10/2009 (1:18 pm)
to be fair... how many RPGs limit to realistic walking speeds, offhand the last one i can think of was an infinity engine game possibly one of the IWD series, pretty much every 3d RPG ive played has a constant movespeed of at least double that of realistic movement speeds.and depending how detailed you want your terrain, a 4096 texture at the highest detail level doesnt really cover much ground and takes a huge level of resources especially if you want a good visibility distance and quality.
On the flip side of that if you dont require high levels of details you can make a 4096 terrain go a LONG way
#17
@Michael -- I do agree that the new terrain system, if you forgive the size restrictions, appears to be easier and better. At the moment, however, I have not been able to do much with opacity layers as I cannot figure out how to get them out of L3dt! I have only just figured out heightfields. Any advice on how I should proceed? Also, what is the "mega terrain trick"? And lastly, if i were to import 9 terrain blocks what exactly would I have? Would I have a game teetering on the edge of disaster or is this a stable solution to my problems? would it be like Everquest where the game paused to load a new zone? Would the exited terrain be overwritten in Ram by the new entered terrain? If these aren't "zones" how could I implement zones?
@Logan and Tom -- I will try out increasing meters per pixel and square size, thanks.
@Jacob -- I tested my 4096 x 4096 both in F11 mode and also in game play mode. It took me only 4 minutes to run from one corner to another using the default avatar and whatever default movement speed he is at in both modes. I think I agree with you that if it took me 35 minutes, I should/would settle for that. Did I do something wrong? Why did it only take me 4 minutes? I did not drop any buildings or other objects to see if the speed seemed absurd, I just crossed an imported heightfield. Also, I see from your posts that you love terrains...if you could point me to any place I could learn a bit more about importing the other layers I need in addition to the heightfield, it would be most appreciated.
@everyone -- thanks for all the info. I do want to be able to have my avatar take one hour from one side of the game world to the other in T3D, whether it be multiple terrain blocks, one terrain, or some type of zoning. I would appreciate anyone's input on how to do so, or pointing me to any tutorials or resources where I can learn how to do so (that are still applicable to T3D!).
05/10/2009 (3:06 pm)
wow, so much info, thanks everyone. I am a bit surprised to learn that the "unlimited terrains" feature of Atlas has been abandoned, even though it never really was unlimited. First let me say that I am a hobbyist so I do struggle with things you all my find easy. With that in mind, this post has raised a few questions in my mind:@Michael -- I do agree that the new terrain system, if you forgive the size restrictions, appears to be easier and better. At the moment, however, I have not been able to do much with opacity layers as I cannot figure out how to get them out of L3dt! I have only just figured out heightfields. Any advice on how I should proceed? Also, what is the "mega terrain trick"? And lastly, if i were to import 9 terrain blocks what exactly would I have? Would I have a game teetering on the edge of disaster or is this a stable solution to my problems? would it be like Everquest where the game paused to load a new zone? Would the exited terrain be overwritten in Ram by the new entered terrain? If these aren't "zones" how could I implement zones?
@Logan and Tom -- I will try out increasing meters per pixel and square size, thanks.
@Jacob -- I tested my 4096 x 4096 both in F11 mode and also in game play mode. It took me only 4 minutes to run from one corner to another using the default avatar and whatever default movement speed he is at in both modes. I think I agree with you that if it took me 35 minutes, I should/would settle for that. Did I do something wrong? Why did it only take me 4 minutes? I did not drop any buildings or other objects to see if the speed seemed absurd, I just crossed an imported heightfield. Also, I see from your posts that you love terrains...if you could point me to any place I could learn a bit more about importing the other layers I need in addition to the heightfield, it would be most appreciated.
@everyone -- thanks for all the info. I do want to be able to have my avatar take one hour from one side of the game world to the other in T3D, whether it be multiple terrain blocks, one terrain, or some type of zoning. I would appreciate anyone's input on how to do so, or pointing me to any tutorials or resources where I can learn how to do so (that are still applicable to T3D!).
#18
We also need paging terrain in our project , and have been advised by Tom Spilman to perhaps integrate Atlas or something similar back into Torque 3d so if you manage it we will buy the code from you , we would do it but we are too busy on other things at the moment
We only need to page 1024 maps at 2m res , or we are going to use the Torque 3d terrain with megaterrain, once we figure it out and break the game up into levels but hide the way they load.
05/10/2009 (3:27 pm)
@BOWe also need paging terrain in our project , and have been advised by Tom Spilman to perhaps integrate Atlas or something similar back into Torque 3d so if you manage it we will buy the code from you , we would do it but we are too busy on other things at the moment
We only need to page 1024 maps at 2m res , or we are going to use the Torque 3d terrain with megaterrain, once we figure it out and break the game up into levels but hide the way they load.
#19
That question is asked the wrong way round then.
The question would in this case boil down to: Depending on if it is meant to run below a GTX260 / HD4850 & 3GB+ RAM installed or not ...
Especially as T3D terrain is much more detailed than the legacy at the same resolution ever was in TGEA / TGE, so at 4096 square you get a really large and detailed terrain, thanks to the new texturing with up to 255 textures and the road & river tool even with more detail than ever before
05/10/2009 (3:29 pm)
Quote:and depending how detailed you want your terrain, a 4096 texture at the highest detail level doesnt really cover much ground and takes a huge level of resources especially if you want a good visibility distance and quality.
That question is asked the wrong way round then.
The question would in this case boil down to: Depending on if it is meant to run below a GTX260 / HD4850 & 3GB+ RAM installed or not ...
Especially as T3D terrain is much more detailed than the legacy at the same resolution ever was in TGEA / TGE, so at 4096 square you get a really large and detailed terrain, thanks to the new texturing with up to 255 textures and the road & river tool even with more detail than ever before
#20
In a 'paging' map system, the better ones i have used, actual keep the player at 0,0,0 and calculate the world around the player. This is not how Torque is designed, so i can not see 'paging' ever working 100% well with Torque.
05/10/2009 (3:34 pm)
I myself, for RPG design, have found a happy place with rather small terrains who are highly detailed, careful 'path' planing and placing common plot details on one terrain in conjunction with loading North/South/East/West maps 'chunks' when the player venture forth, works great. It is also much easier then designing ONE BIGASS map. My play-testers never complain unless the load times are more then 50 seconds, or when they need to go back and forth often.In a 'paging' map system, the better ones i have used, actual keep the player at 0,0,0 and calculate the world around the player. This is not how Torque is designed, so i can not see 'paging' ever working 100% well with Torque.
Associate Tom Spilman
Sickhead Games
If you really wanted to try it you could patch up a few little places where it blows out expected limits, but its not something i think we'll support officially.
Really this is a bug... it should keep you from importing one bigger than that in the first place.