Upcoming(?) Terrain improvements
by Gerry "Hermetic" Smith · in Torque 3D Professional · 03/04/2010 (3:14 pm) · 20 replies
Well instead of taking away from more specific issues, in threads where these kinds of things have come up, maybe I should start a new thread.
You can see some of this talk in these two threads, and it has come up in other places but I won't dig up each and every post where it has occurred:
T3D terrain texture blend
Terrain Tile
The question I, and others, have been asking is one about the general direction of terrain improvements, and whether indeed there are plans for further improvements in this area.
Tom told us in Sept. last year how they "have alot of really cool stuff coming up" and "whatever isn't done in 1.1 will get moved into 1.2." Even though 1.1 is still in beta it is supposedly feature complete, and so first off maybe a Torque employee, and/or associate, and/or somebody 'in the know', can point out what improvements, if any, has been added so far. And secondly maybe one of these same people can give us a hint of what we might expect in 1.2 and later versions?
I for one would really like to see some kind of terrain, and asset, streaming/paging system. I have seen much interest for this, even among the Sickhead crew(I am pretty sure).
I first got into Torque Tech and bought the TGEA engine because of what I heard about Atlas terrain, but it was dropped out because it caused more problems than it solved supposedly(I don't know much about it since I didn't have much time to try it out). And I became an early adopter of T3D because of the promise of an even better terrain system. No doubt there have been improvements already but perhaps not really in the areas that got me excited about Atlas; the streaming of large worlds kinda thing...
So this is where I am coming from and shows what kind of info that would really help me in my own plans to move forward.
Others can feel free to express what they would like to see in terms of terrain improvements of course, but really I am asking the powers that be if they could please give us some ideas of what we can expect in these areas... pretty please? ( I promise I will do what I can to have my country's hockey team stop beating up on your country's hockey team. )
You can see some of this talk in these two threads, and it has come up in other places but I won't dig up each and every post where it has occurred:
T3D terrain texture blend
Terrain Tile
The question I, and others, have been asking is one about the general direction of terrain improvements, and whether indeed there are plans for further improvements in this area.
Tom told us in Sept. last year how they "have alot of really cool stuff coming up" and "whatever isn't done in 1.1 will get moved into 1.2." Even though 1.1 is still in beta it is supposedly feature complete, and so first off maybe a Torque employee, and/or associate, and/or somebody 'in the know', can point out what improvements, if any, has been added so far. And secondly maybe one of these same people can give us a hint of what we might expect in 1.2 and later versions?
I for one would really like to see some kind of terrain, and asset, streaming/paging system. I have seen much interest for this, even among the Sickhead crew(I am pretty sure).
I first got into Torque Tech and bought the TGEA engine because of what I heard about Atlas terrain, but it was dropped out because it caused more problems than it solved supposedly(I don't know much about it since I didn't have much time to try it out). And I became an early adopter of T3D because of the promise of an even better terrain system. No doubt there have been improvements already but perhaps not really in the areas that got me excited about Atlas; the streaming of large worlds kinda thing...
So this is where I am coming from and shows what kind of info that would really help me in my own plans to move forward.
Others can feel free to express what they would like to see in terms of terrain improvements of course, but really I am asking the powers that be if they could please give us some ideas of what we can expect in these areas... pretty please? ( I promise I will do what I can to have my country's hockey team stop beating up on your country's hockey team. )
#2
I think first of all, it has to be fixed.
And i will appreciate a lot - terrain blocks support, but it cannot be rightly done without the bug fixing.
03/05/2010 (6:04 am)
I vote for streaming/paging system for SceneObjects because we are using big terrains 20x20 km and really need it. And terrain improvements at all, see picture(3 bug terrain edge shaded, Forest editor and ground cover lost terrain bounds together).I think first of all, it has to be fixed.
And i will appreciate a lot - terrain blocks support, but it cannot be rightly done without the bug fixing.
#3
Given what has been said about what we might expect to see in coming versions (and I don't take any of it as written-in-stone promises that we should get upset over if they don't end up coming to fruition in exactly the way we would like to see) and given that it is almost half a year later and version 1.1 is upon us, albeit in beta, but still, supposedly, feature complete, then I don't see how my questions are unreasonable or unfair.
03/05/2010 (8:41 am)
Quote:Not many games I have played use paging terrain.Does that mean it can't be done? Does that mean it shouldn't be done? Does that mean you feel this game engine is only meant to make games that have already been played? No disrespect but I am more interested in games you, and others, will play. Does that mean I have chosen the wrong engine for what I want to do? That is what I am trying to determine...
Quote:Not sure if thats the highest on there list right now.If that is what they have decided then so be it. I will adjust my plans accordingly. I am not making demands. I am just looking for answers. Even an answer of, "We have made no plans one way or the other in regard to terrain," while disappointing, would still be something. Any answer would be better than being kept in the dark.
Given what has been said about what we might expect to see in coming versions (and I don't take any of it as written-in-stone promises that we should get upset over if they don't end up coming to fruition in exactly the way we would like to see) and given that it is almost half a year later and version 1.1 is upon us, albeit in beta, but still, supposedly, feature complete, then I don't see how my questions are unreasonable or unfair.
#4
03/05/2010 (8:59 am)
Even Unity has streaming or paged assets and terrain (whatever the proper term is) don't they? I think it would expand T3D's adaptability for MMO games. As I understand it, multiplayer games in general and MMO games specifically are the last bastion of PC superiority over the consoles. 
#5
03/05/2010 (9:03 am)
im all for paging terrain. the more features the better IMO. i dont think games like WOW would be less enjoyable if it had more zones. dont mind zoning, but some do. sometimes all planned features will not make a game. i would love to have streaming terrain in our MMO, but having zones is not gamebreaking.
#6
Or we could create cool 3D scenes for small cottage town or whatever 3D reels in Torque with cars, walking people and so on, it is 10x10 rm as minimum with 0,5 meters grid.
It can really expand T3D's adaptability as game engine and not only!
03/05/2010 (9:11 am)
And Torque3D could be used not only for games, i am trying to create military simulator with huge terrains support.Or we could create cool 3D scenes for small cottage town or whatever 3D reels in Torque with cars, walking people and so on, it is 10x10 rm as minimum with 0,5 meters grid.
It can really expand T3D's adaptability as game engine and not only!
#7
That's good to know. Even if Unity didn't have that GG shouldn't want to limit itself to only what has already been done by somebody else.
03/05/2010 (9:12 am)
Quote:Even Unity has streaming or paged assets and terrain...
That's good to know. Even if Unity didn't have that GG shouldn't want to limit itself to only what has already been done by somebody else.
#8
There has been a huge thread on this before, and I'll just say what I know from experience: don't expect paging terrains to make a comeback anytime soon, here's why:
1) "Paging terrain" alone does not allow you to make large worlds (the main purpose for a paging terrain). What about the objects on top of the terrain? Also, how would this work in a server-client situation? The server would still need all terrain pages to be loaded, because different clients could be anywhere and there. So a paging terrain is useless unless you can page everything.
2) The T3D resource manager doesn't support asynchronous loading of assets, and it would require a major refactoring of pretty much *all* classes that load resources for it to be possible. Even if you make terrains page their textures (so you can use several terrains stitched together without blowing up VRAM), there'll be hiccups while textures are loaded from disk->RAM->VRAM.
03/05/2010 (9:32 am)
@Steve: you are mistaken. Unity supports web streaming of assets (in the Pro version). Not MMO-level RAM paging of objects.There has been a huge thread on this before, and I'll just say what I know from experience: don't expect paging terrains to make a comeback anytime soon, here's why:
1) "Paging terrain" alone does not allow you to make large worlds (the main purpose for a paging terrain). What about the objects on top of the terrain? Also, how would this work in a server-client situation? The server would still need all terrain pages to be loaded, because different clients could be anywhere and there. So a paging terrain is useless unless you can page everything.
2) The T3D resource manager doesn't support asynchronous loading of assets, and it would require a major refactoring of pretty much *all* classes that load resources for it to be possible. Even if you make terrains page their textures (so you can use several terrains stitched together without blowing up VRAM), there'll be hiccups while textures are loaded from disk->RAM->VRAM.
#9
If they can't, or don't want to, for whatever reason, implement something like that then it would be nice to hear them say that. Also it would be nice to hear them describe what "really cool stuff" and improvements they 'do' have in mind for terrain. I tend to agree with James B. that terrain is perhaps the biggest weakness of this engine.
03/05/2010 (9:57 am)
I for one, while I am interested in rpgs, I am not interested in making an mmo. Quote:There has been a huge thread on this before...Yes, and your two points have been commented on by me and others there, which is why this thread isn't about streaming terrain and assets alone but asking what plans they do have.
If they can't, or don't want to, for whatever reason, implement something like that then it would be nice to hear them say that. Also it would be nice to hear them describe what "really cool stuff" and improvements they 'do' have in mind for terrain. I tend to agree with James B. that terrain is perhaps the biggest weakness of this engine.
#10
And i agree, Road map at least for terrain improvements will be great to see.
Now i know T3D bug tracking system is on the way to us, maybe we will see road map for future releases soon too.
03/05/2010 (10:08 am)
Tom promised support for multiply Terrain Blocks in the near future. This along with bug fixing could be great first steps i think. And i agree, Road map at least for terrain improvements will be great to see.
Now i know T3D bug tracking system is on the way to us, maybe we will see road map for future releases soon too.
#11
While I can see some reasons this might be useful, namely to create higher-resolution terrain by using several smaller terrain blocks instead of 1 large one, there's an issue with massive scale environments that everyone seems to neglect over and over:
32-bit floating point precision does not allow for objects to reasonably exist at coordinates higher than about 10-20,000 units. Beyond this point, objects begin to "jitter" as their physics and position calculations don't have enough decimal precision to render smoothly.
In fact, parts of objects begin to jitter relative to each other (by parts I mean individual nodes in the model itself). Z-buffering of decals breaks down. The rendering system itself begins to fail (D3D, of course, renders in 32-bit float).
So basically, unless you intend to solve this problem (see: paging/relative scenegraph), huge terrains tend to be a wasted effort. Though I suppose you could still use them for views of distant environments, or in singleplayer by simply recentering the world everytime the player moves past 10k coords (very doable in singleplayer -- much harder in multiplayer).
For anyone who thinks this is a Torque-specific limitation, think again. Any game that's handling massive environments -- and I actually can't think of any which are truly massive, the GTA games would all fit into T3D limits -- is likely using some kind of paging/offset system to bypass this issue.
If your concern is just being able to see into the distance, not actually travel there, then you should look into some lower-cost solutions. Terrain that your players can't actually reach doesn't need to be rendered with the same detail as your "playable" terrain, and can be handled with any number of "background" solutions (including a relatively low-res giant second terrain object).
RE: MMOs:
The old standby, eh? Only 1 commercial MMO has ever been created with Torque, and it didn't use streaming/paging terrain or massive seamless environments. It was built on solid game design and technical know-how.
As far as MMOs being the only advantage PCs have over consoles? I want to break that statement down with crowbars and fire, but in the interest of civility, I won't. Suffice it to say, serious PC gamers are generally annoyed that forced console equivalence is holding PC games back -- many games get scaled back in size and complexity so they'll run on a 360 or PS3 or better appeal to the console audience. Additionally, we've lost entire genres to console compatibility (RTS, TBS, RPG, etc).
Moving on from the ranting portion of the program I'd also be interested to hear more about upcoming features. I do understand why the devs might be less than thrilled about giving us their current 2-year plan, though.
03/05/2010 (11:53 am)
RE: Streaming/paging terrain:While I can see some reasons this might be useful, namely to create higher-resolution terrain by using several smaller terrain blocks instead of 1 large one, there's an issue with massive scale environments that everyone seems to neglect over and over:
32-bit floating point precision does not allow for objects to reasonably exist at coordinates higher than about 10-20,000 units. Beyond this point, objects begin to "jitter" as their physics and position calculations don't have enough decimal precision to render smoothly.
In fact, parts of objects begin to jitter relative to each other (by parts I mean individual nodes in the model itself). Z-buffering of decals breaks down. The rendering system itself begins to fail (D3D, of course, renders in 32-bit float).
So basically, unless you intend to solve this problem (see: paging/relative scenegraph), huge terrains tend to be a wasted effort. Though I suppose you could still use them for views of distant environments, or in singleplayer by simply recentering the world everytime the player moves past 10k coords (very doable in singleplayer -- much harder in multiplayer).
For anyone who thinks this is a Torque-specific limitation, think again. Any game that's handling massive environments -- and I actually can't think of any which are truly massive, the GTA games would all fit into T3D limits -- is likely using some kind of paging/offset system to bypass this issue.
If your concern is just being able to see into the distance, not actually travel there, then you should look into some lower-cost solutions. Terrain that your players can't actually reach doesn't need to be rendered with the same detail as your "playable" terrain, and can be handled with any number of "background" solutions (including a relatively low-res giant second terrain object).
RE: MMOs:
The old standby, eh? Only 1 commercial MMO has ever been created with Torque, and it didn't use streaming/paging terrain or massive seamless environments. It was built on solid game design and technical know-how.
As far as MMOs being the only advantage PCs have over consoles? I want to break that statement down with crowbars and fire, but in the interest of civility, I won't. Suffice it to say, serious PC gamers are generally annoyed that forced console equivalence is holding PC games back -- many games get scaled back in size and complexity so they'll run on a 360 or PS3 or better appeal to the console audience. Additionally, we've lost entire genres to console compatibility (RTS, TBS, RPG, etc).
Moving on from the ranting portion of the program I'd also be interested to hear more about upcoming features. I do understand why the devs might be less than thrilled about giving us their current 2-year plan, though.
#12
On the contrary. I see the precision issue pointed out every time the streaming idea comes up. And you yourself even touched on some possible solutions. And I appreciate the views, and warnings, of people like you and Manoel, even though it can be like a splash of cold water on my dreams... ;)
And even though none of this stuff would be easy, doesn't in itself mean it wouldn't be worthwhile. The fact that I see people who know a lot more about this engine than I, like some of the Sickhead crew and Ted S. etc., who not only seem to think it is doable but something they would like to see in the engine, gives me hope. I might even like to have a go at it myself if I could find out if GG isn't planning on such a thing. Please GG let us know your plans so we can make plans of our own.
Because they were less than successful in the past? So they should just give up on trying to keep their members and potential customers informed? I don't really understand this. As long as they are careful about what they say so as not to mislead people. As long as they keep to that clarity and are quick to respond with explanations, and even apologize if necessary, if things don't go the way they planned. If they let users and potential customers feel their input is appreciated, and address their questions and concerns in a timely and respectful manner, and keep members in-the-loop, I feel any problems which may arise from miscommunications will be far less than the frustration and anger which arises from users being kept in the dark.
People can be very forgiving if an effort is made, and seen to be made, by a company seeking to do the right thing for its customers in terms of keeping them up-to-speed and well informed. Customers can be made to feel a part of the process instead of feeling like they have to constantly beg for a bone to be thrown their way. I feel a commitment to the membership like that would go a long ways to Eric's stated goal of retention, if I am reading him right in that other thread.
edit/ to paraphrase a bit of ancient wisdom: "It is better to light a candle then to [have your customers] curse [you out for] the darkness."
03/05/2010 (12:47 pm)
Quote:there's an issue with massive scale environments that everyone seems to neglect over and over...
On the contrary. I see the precision issue pointed out every time the streaming idea comes up. And you yourself even touched on some possible solutions. And I appreciate the views, and warnings, of people like you and Manoel, even though it can be like a splash of cold water on my dreams... ;)
And even though none of this stuff would be easy, doesn't in itself mean it wouldn't be worthwhile. The fact that I see people who know a lot more about this engine than I, like some of the Sickhead crew and Ted S. etc., who not only seem to think it is doable but something they would like to see in the engine, gives me hope. I might even like to have a go at it myself if I could find out if GG isn't planning on such a thing. Please GG let us know your plans so we can make plans of our own.
Quote:I do understand why the devs might be less than thrilled about giving us their current 2-year plan, though.
Because they were less than successful in the past? So they should just give up on trying to keep their members and potential customers informed? I don't really understand this. As long as they are careful about what they say so as not to mislead people. As long as they keep to that clarity and are quick to respond with explanations, and even apologize if necessary, if things don't go the way they planned. If they let users and potential customers feel their input is appreciated, and address their questions and concerns in a timely and respectful manner, and keep members in-the-loop, I feel any problems which may arise from miscommunications will be far less than the frustration and anger which arises from users being kept in the dark.
People can be very forgiving if an effort is made, and seen to be made, by a company seeking to do the right thing for its customers in terms of keeping them up-to-speed and well informed. Customers can be made to feel a part of the process instead of feeling like they have to constantly beg for a bone to be thrown their way. I feel a commitment to the membership like that would go a long ways to Eric's stated goal of retention, if I am reading him right in that other thread.
edit/ to paraphrase a bit of ancient wisdom: "It is better to light a candle then to [have your customers] curse [you out for] the darkness."
#13
Technology changes, company goals change, and often great ideas end up being technically impossible or unworkable when you actually get to the nuts and bolts of them (see: Atlas). And more often than not, the plan is "explore what's actually possible and try to improve the feature somehow." That's not a very reassuring todo list to give your users. Sometimes, even when the long-term path to a goal is clear, the powers that be (execs, team leads, whatever) actually don't want this stuff out there due to competition issues (not saying this attitude is right, simply that it's not uncommon).
I think the issue here is that a lot of us (you included obviously) would be much happier to be part of a discussion on potentially vaporware features than simply be left out of it entirely. Unfortunately there's also a large group of users who don't get how things work in software development and will spend the next year writing angry letters about how SuperTerrainX17 wasn't exactly what they expected, and demanding to know why they were lied to.
Personally, I'd vote for an open discussion on the team's goals, but maybe not explicitly how these goals will be met. That said, I still get why it's a tough situation for the dev team.
03/08/2010 (12:37 pm)
I actually said that because I have experience with being on the other side of this issue. Users can be demanding and difficult about these things, and may actually be more upset if the announced plans change (and they always do) than they would have been knowing nothing.Technology changes, company goals change, and often great ideas end up being technically impossible or unworkable when you actually get to the nuts and bolts of them (see: Atlas). And more often than not, the plan is "explore what's actually possible and try to improve the feature somehow." That's not a very reassuring todo list to give your users. Sometimes, even when the long-term path to a goal is clear, the powers that be (execs, team leads, whatever) actually don't want this stuff out there due to competition issues (not saying this attitude is right, simply that it's not uncommon).
I think the issue here is that a lot of us (you included obviously) would be much happier to be part of a discussion on potentially vaporware features than simply be left out of it entirely. Unfortunately there's also a large group of users who don't get how things work in software development and will spend the next year writing angry letters about how SuperTerrainX17 wasn't exactly what they expected, and demanding to know why they were lied to.
Personally, I'd vote for an open discussion on the team's goals, but maybe not explicitly how these goals will be met. That said, I still get why it's a tough situation for the dev team.
#14
And as I've tried to point out... I can accept that, if that is the answer. Being totally ignored and kept in the dark and given no answer whatsoever doesn't inspire much confidence about the direction things are going with this company, and even if some wonderful gotta-have-it tech was introduced into the engine I will still be reluctant to lay out even more money for the next paid upgrade. I have already spent a large amount of money on this engine so I will wait to see what happens in the meantime...
I've been into software dev, and business in general, for decades so I don't think this is coming from some impatient kid - or someone who only knows things from the customers perspective - with unrealistic ideas about how things work and how customers should be treated. I don't think most of the frustration and complaints I see here can be dismissed so easily.
03/08/2010 (1:32 pm)
Quote:Technology changes, company goals change, and often great ideas end up being technically impossible or unworkable...
And as I've tried to point out... I can accept that, if that is the answer. Being totally ignored and kept in the dark and given no answer whatsoever doesn't inspire much confidence about the direction things are going with this company, and even if some wonderful gotta-have-it tech was introduced into the engine I will still be reluctant to lay out even more money for the next paid upgrade. I have already spent a large amount of money on this engine so I will wait to see what happens in the meantime...
I've been into software dev, and business in general, for decades so I don't think this is coming from some impatient kid - or someone who only knows things from the customers perspective - with unrealistic ideas about how things work and how customers should be treated. I don't think most of the frustration and complaints I see here can be dismissed so easily.
#15
1.1 does not have official support for multiple terrain blocks (there are known issues here). Tom, however, does have plans to get back to this for 1.2 and make it officially support multiple terrain blocks per level as well as giving it tools to work with them.
As for streaming, I don't see this happening for 1.2 primarily because it appears to me that there are fundamental architectural problems that need to be solved first and solved thoroughly. Torque as a codebase already has a proud number of years on its back and in the field of software engineering this implies that there is a legacy being carried around--a legacy which requires re-engineering parts of the codebase to bring it in line with changes that have happened.
This is currently well in progress and to my delight, I see legacy Torque code (of which I am/were for a significant part not so fond of as it had often been falling significantly short of modern engineering practices) replaced, rewritten, or becoming obsolete at a solid pace here in the dev team. I am confident that by the end of this year, we have worked off a huge part of this debt.
Oh... hmm... back on topic...
I had implemented a resource streaming system in Torque some time ago and, like Manoel already stated here before, it required resource-related code to be brought over to the new system. It already worked nicely though with TerrainBlocks and just maybe I'll take a look at this again when 1.2 comes around.
03/09/2010 (8:53 pm)
I refrained from participating in the discussion so far since this isn't really my area of the codebase and if I do say something now, please take it with caution and as 100% non-confirmed-and-no-promises whatsoever.1.1 does not have official support for multiple terrain blocks (there are known issues here). Tom, however, does have plans to get back to this for 1.2 and make it officially support multiple terrain blocks per level as well as giving it tools to work with them.
As for streaming, I don't see this happening for 1.2 primarily because it appears to me that there are fundamental architectural problems that need to be solved first and solved thoroughly. Torque as a codebase already has a proud number of years on its back and in the field of software engineering this implies that there is a legacy being carried around--a legacy which requires re-engineering parts of the codebase to bring it in line with changes that have happened.
This is currently well in progress and to my delight, I see legacy Torque code (of which I am/were for a significant part not so fond of as it had often been falling significantly short of modern engineering practices) replaced, rewritten, or becoming obsolete at a solid pace here in the dev team. I am confident that by the end of this year, we have worked off a huge part of this debt.
Oh... hmm... back on topic...
I had implemented a resource streaming system in Torque some time ago and, like Manoel already stated here before, it required resource-related code to be brought over to the new system. It already worked nicely though with TerrainBlocks and just maybe I'll take a look at this again when 1.2 comes around.
#16
03/09/2010 (10:29 pm)
I'll make sure that at the least you can use multiple TerrainBlocks in 1.2 and some GUI for simple placement of them. What you won't get that round is a solution for paging terrain or dealing with precision issues... no timeline on that, but we all want to see it happen.
#17
03/10/2010 (4:06 am)
Some solution for tiling here: www.torquepowered.com/community/blogs/view/19451
#18
03/10/2010 (8:36 am)
It's good to know we terrain freaks have something to look forward to :-) . Thanks for the info.
#19
03/10/2010 (10:57 am)
Thank you. This information is helpful. 
#20
__________
james
03/14/2010 (1:42 pm)
I dunno if it's been covered, but to clarify an earlier post: no, Unity does NOT support terrain paging. Especially internally. There are a few user attempts at a scripted solution, but it has been reported that the terrain engine in Unity has a prominent memory leak that will cause hard crashes.__________
james
Torque 3D Owner Stadi Thompson