TGE, TGEA, T3D... Atlas Removal .. Should I be worried??
by Eamonn Glass · in Torque Game Engine Advanced · 01/30/2009 (10:58 am) · 14 replies
Hey Folks,
I am researching engines. Torque looks promising. But I have some significant concerns, wondering if folks can assist.
1. New engines faster than the community resources can catch up.
One of the strong points of Torque is the community and the resources that community has spawned and shared (either via kindness or the commercial marketplace). But I see a LOT of these that are still TGE (or even specific versions of TGE). It seems as though TGE has been deprecated by TGEA. It appears as though some of these resources will be ported to TGEA about the time T3D is released. Or even after.. or in some cases the resources will die on the vine.
My concern is that the community resources aspect of Torque (free or commercial), stuff that helps new game developers bootstrap their project, is not keeping up with the engine development cycle and, as a result, the state of community resources available is not as robust as it looks on the surface.
2. Atlas Removal in T3D
This is a really huge concern for me and more of an observation that a really specific question. To add in a new terrain engine, and then remove it one generation later... how did that happen? Was the feature so under-designed in the first place that the limitations were not identified? And when identified they were so severe that the feature had to be removed? How was something like that missed when adding in the new terrain engine?
Are there other parts of the engine that might see the same fate? I can understand legacy code being re-factored causing complete subsystem redesign, and as a result breaking backward compatibility. But Atlas was new in TGEA less than 2 years ago. The addition and removal of that is a concern from the perspective of the "design debt" that was not paid by the engine developers being paid by the engine consumers.
3. What IS the T3D solution for LARGE terrains?
The project I would be doing on Torque requires as large a terrain world as we can get with very high view distances and still maintain good performance. We'd like to see 8km x 8km terrains with 1500m view distances and still have the ability to have good looking models and ~130 players in the world concurrently, possibly with additional AI bots on top of that.
Atlas sounded promising when combined with the reportedly outstanding TGEA netcode. What will be the solution for this sort of requirements in T3D? Or are these requirements simply not feasible (either in TGEA or T3D)?
Thanks in advance for any help provided.
I am researching engines. Torque looks promising. But I have some significant concerns, wondering if folks can assist.
1. New engines faster than the community resources can catch up.
One of the strong points of Torque is the community and the resources that community has spawned and shared (either via kindness or the commercial marketplace). But I see a LOT of these that are still TGE (or even specific versions of TGE). It seems as though TGE has been deprecated by TGEA. It appears as though some of these resources will be ported to TGEA about the time T3D is released. Or even after.. or in some cases the resources will die on the vine.
My concern is that the community resources aspect of Torque (free or commercial), stuff that helps new game developers bootstrap their project, is not keeping up with the engine development cycle and, as a result, the state of community resources available is not as robust as it looks on the surface.
2. Atlas Removal in T3D
This is a really huge concern for me and more of an observation that a really specific question. To add in a new terrain engine, and then remove it one generation later... how did that happen? Was the feature so under-designed in the first place that the limitations were not identified? And when identified they were so severe that the feature had to be removed? How was something like that missed when adding in the new terrain engine?
Are there other parts of the engine that might see the same fate? I can understand legacy code being re-factored causing complete subsystem redesign, and as a result breaking backward compatibility. But Atlas was new in TGEA less than 2 years ago. The addition and removal of that is a concern from the perspective of the "design debt" that was not paid by the engine developers being paid by the engine consumers.
3. What IS the T3D solution for LARGE terrains?
The project I would be doing on Torque requires as large a terrain world as we can get with very high view distances and still maintain good performance. We'd like to see 8km x 8km terrains with 1500m view distances and still have the ability to have good looking models and ~130 players in the world concurrently, possibly with additional AI bots on top of that.
Atlas sounded promising when combined with the reportedly outstanding TGEA netcode. What will be the solution for this sort of requirements in T3D? Or are these requirements simply not feasible (either in TGEA or T3D)?
Thanks in advance for any help provided.
#2
The benefit of not using Atlas for a (comparatively) small terrain like that is that you'll then be able to edit it in engine.
01/30/2009 (11:36 am)
As Ted says, Atlas was pretty difficult to work with. The pipeline compared to working with Legacy terrain was pretty terrible. Most likely we'll see some sort of improved version of the "MegaTerrain" concept. I don't really know what's going to happen on this front though, so take anything I say with a grain of salt. Whatever happens, I think you'll probably be able to do what you need to with it (you would be able to do an 8km x 8km terrain right now with a MegaTerrain with a square size of 16m, but that's not ideal). If Legacy were extended to do larger base sized terrains, you could easily get an 8km map with a square size of 8m.The benefit of not using Atlas for a (comparatively) small terrain like that is that you'll then be able to edit it in engine.
#3
I don't really know what they have in store for the new Megaterrains, but in another thread they threw out a number of about twice the size of the current ones. That would be sufficient for most game types, unless your doing a flight simulator. In that case though you could increase the tile size to a larger number and get away with blurriness in your textures because noone would ever be that close to the terrain. If they can get the performance of Legacy up to snuff, and deliver decent looking textures up close, the its probably the right move to make. Being able to use things like the TWSurfaceReference would make it worth it, and Legacy in general is easier to extend than Atlas was.
01/30/2009 (10:37 pm)
I got used to using Grome to do my editing with Atlas, though being able to tweak terrain in the game would definately be a plus. Depending on how the new terrain is handled, this may be a good thing though. It was impossible to get a reasonable texel size on your terrain with without using a really small tile size, which would then dramatically reduce your overall terrain size. We wound up having to write some code to use two different detail textures based on where you were at, and only using atlas to do the coloring underneath the detail textures, where the detail textures provided most of the look. That came at a performance hit. I don't really know what they have in store for the new Megaterrains, but in another thread they threw out a number of about twice the size of the current ones. That would be sufficient for most game types, unless your doing a flight simulator. In that case though you could increase the tile size to a larger number and get away with blurriness in your textures because noone would ever be that close to the terrain. If they can get the performance of Legacy up to snuff, and deliver decent looking textures up close, the its probably the right move to make. Being able to use things like the TWSurfaceReference would make it worth it, and Legacy in general is easier to extend than Atlas was.
#4
01/31/2009 (4:02 pm)
I am not sure GG can replace the capabilities that are in Grome for building Terrains. But, Whatever they do you will still be able to create terrains using Grome, I have recently bought it and was shocked to hear the news from GG a few weeks later. That type stuff is what should be in a roadblock so people can plan ahead. 
#5
This has been a pretty big issue for some people in the past. The reality is that not everybody interested in using some of the resources posted here are going to have the ability to re-implement them in an updated engine. It may even get worse with the significant refactor that Torque 3D will no doubt be undergoing. However, I don't really see this as a significant issue, or at least any more significant than it is now.
TGE and TGEA will still be used for a long time to come. They are both awesome engines to use and thus the current resources will still remain relevant. Most of the popular resources will be upgraded and updated (as they are now), but the smaller, more obscure resources will probably remain as they are.
It takes a pretty big man to stand up and admit that they did something wrong. The reality is the Atlas was never fully supported, it has always had issues. I am sure that it isn't for a lack of trying either. Terrains are important, no doubt about it, but they are one aspect of the engine and spending months of development time in fixing the issues may not be worth it.
Overall I think that Torque 3D (and GarageGames) has a great future, there are some fantastic things happening and there are some incredibly talented people working there. I cannot wait to read more blogs from Brett, I love it.
01/31/2009 (4:25 pm)
Quote:1. New engines faster than the community resources can catch up.
This has been a pretty big issue for some people in the past. The reality is that not everybody interested in using some of the resources posted here are going to have the ability to re-implement them in an updated engine. It may even get worse with the significant refactor that Torque 3D will no doubt be undergoing. However, I don't really see this as a significant issue, or at least any more significant than it is now.
TGE and TGEA will still be used for a long time to come. They are both awesome engines to use and thus the current resources will still remain relevant. Most of the popular resources will be upgraded and updated (as they are now), but the smaller, more obscure resources will probably remain as they are.
Quote:2. Atlas Removal in T3D
It takes a pretty big man to stand up and admit that they did something wrong. The reality is the Atlas was never fully supported, it has always had issues. I am sure that it isn't for a lack of trying either. Terrains are important, no doubt about it, but they are one aspect of the engine and spending months of development time in fixing the issues may not be worth it.
Overall I think that Torque 3D (and GarageGames) has a great future, there are some fantastic things happening and there are some incredibly talented people working there. I cannot wait to read more blogs from Brett, I love it.
#6
Personally, I am glad they have dropped it. A system that has many of the benefits of Atlas, but is as easy to use as legacy terrain, seems much more promising.
01/31/2009 (6:32 pm)
As I recall, and I could be way off here, Atlas was developed independently by Ben Garney, and later added to the TGEA package only because it was kinda cool and people liked it. As Phillip said, it was never fully supported.Personally, I am glad they have dropped it. A system that has many of the benefits of Atlas, but is as easy to use as legacy terrain, seems much more promising.
#7
I think what I would be looking for, and I hope GG is planning is to implement something into TGEA that will make upgrading to T3D later easier.
I was just getting started using Atlas, and I have not run into any issues, other than a simple build issue.
But in the area of terrain, I really don't care much myself as I am no longer considering large terrain areas.
01/31/2009 (6:45 pm)
I agree...I think what I would be looking for, and I hope GG is planning is to implement something into TGEA that will make upgrading to T3D later easier.
I was just getting started using Atlas, and I have not run into any issues, other than a simple build issue.
But in the area of terrain, I really don't care much myself as I am no longer considering large terrain areas.
#8
I'm working with Atlas since our worlds need to be around 20km x 20km physically - we are working on simulations.
One feature we absolutely need is the import function for pictures (e.g. grayscale) as heightmaps. This was both possible for legacy terrain and Atlas. Especially for ATLAS, L3DT created very beautiful landscapes - on the other hand, it is not easy to get the exported Atlas terrain running...
Is there anything known about the terrain and landscape creation in T3D? I mean, will it be possible to import picture-based heightmaps, and will there be the possibility to use programs like L3DT or Terragen to create heightmaps and textures?
02/01/2009 (3:55 am)
Hi,I'm working with Atlas since our worlds need to be around 20km x 20km physically - we are working on simulations.
One feature we absolutely need is the import function for pictures (e.g. grayscale) as heightmaps. This was both possible for legacy terrain and Atlas. Especially for ATLAS, L3DT created very beautiful landscapes - on the other hand, it is not easy to get the exported Atlas terrain running...
Is there anything known about the terrain and landscape creation in T3D? I mean, will it be possible to import picture-based heightmaps, and will there be the possibility to use programs like L3DT or Terragen to create heightmaps and textures?
#9
While I think it's justifiable to say something like this "takes courage", this is after all a business. I think more than courage, I'd appreciate a detailed explanation for the rationale of the removal as well as clear insight into the future direction for terrain related features.
To have to pick something like this out of a thread is a bit of a concern in my view. I would see this as a strategic level decision that needs more attention.
==
I am still not clear what the approach for large terrains is in T3D and given the Atlas snippet, I would hope there is some more detailed information forthcoming. I have no idea what the timeline for T3D is, but I would hope there is some clarity around this issue fairly soon.
==
wrt the issue about community resources and their deprecation by later releases of the engine... well.. I think a strength of the GG products is the community support and resources. I think GG could apply a concerted effort at making these resources easier to determine compatibility with current / future revisions. I mean even as simple as a "status block" on each resource that can help determine if something from 2002 still works on a 2009 era engine.
But on a broader note, I think this is a risk for GG that they need to manage carefully.
Thanks for the answers Folks.
02/02/2009 (1:55 pm)
I can't say I get the "warm and fuzzies" regarding the addition and removal of a terrain engine over a single generation of the engine. While I think it's justifiable to say something like this "takes courage", this is after all a business. I think more than courage, I'd appreciate a detailed explanation for the rationale of the removal as well as clear insight into the future direction for terrain related features.
To have to pick something like this out of a thread is a bit of a concern in my view. I would see this as a strategic level decision that needs more attention.
==
I am still not clear what the approach for large terrains is in T3D and given the Atlas snippet, I would hope there is some more detailed information forthcoming. I have no idea what the timeline for T3D is, but I would hope there is some clarity around this issue fairly soon.
==
wrt the issue about community resources and their deprecation by later releases of the engine... well.. I think a strength of the GG products is the community support and resources. I think GG could apply a concerted effort at making these resources easier to determine compatibility with current / future revisions. I mean even as simple as a "status block" on each resource that can help determine if something from 2002 still works on a 2009 era engine.
But on a broader note, I think this is a risk for GG that they need to manage carefully.
Thanks for the answers Folks.
#10
It is bad if you want to edit terrain in game, and it is not great for downloadable games because large terrains use lots of disk space.
In other words, it is fantastic for simulations, but not very good for games. The Torque team is focusing on supporting games - small to medium scale games - and has made their technology decisions appropriately. Parts of Atlas have gone on to be used elsewhere in the engine, and research I did while working on it has been picked up for future releases of the engine.
When I started on Atlas, everyone said - "make it support really big terrains." But it turned out what everyone wanted, once they got that, was really easy to edit terrains. Future terrain system developers take heed! :)
02/26/2009 (7:44 pm)
Atlas is good for large datasets, importing stuff from GIS or heightmaps or L3DT etc. It supports long view distances and with a little care high texel densities. (At least it did when I stopped working on it.)It is bad if you want to edit terrain in game, and it is not great for downloadable games because large terrains use lots of disk space.
In other words, it is fantastic for simulations, but not very good for games. The Torque team is focusing on supporting games - small to medium scale games - and has made their technology decisions appropriately. Parts of Atlas have gone on to be used elsewhere in the engine, and research I did while working on it has been picked up for future releases of the engine.
When I started on Atlas, everyone said - "make it support really big terrains." But it turned out what everyone wanted, once they got that, was really easy to edit terrains. Future terrain system developers take heed! :)
#11
Can you recommend sites/books/whitepapers that would give me a good understanding of how Atlas works (in theory & practice)? I've read the code, but I think I need a higher level overview.
The lack of in-game editing capabilities & large file size, are not an issue for my plans, so I intend to 'carry the torch' (so to speak) for Atlas.
02/27/2009 (6:35 am)
@BenCan you recommend sites/books/whitepapers that would give me a good understanding of how Atlas works (in theory & practice)? I've read the code, but I think I need a higher level overview.
The lack of in-game editing capabilities & large file size, are not an issue for my plans, so I intend to 'carry the torch' (so to speak) for Atlas.
#13
Read the SGI Clipmap paper for details on how the clipmap system works. I also wrote an article for Game Programming Gems 7 on the technique.
There are some docs that I wrote explaining how the system works, but I honestly don't know where they would be now - they should be included in the distribution or up on TDN. They covered the script commands, general overview, etc.
The best guide is the source code, at least as far as the internals go. Every class should be documented pretty thoroughly with Doxygen.
Basically you have multiple quadtrees of data, for geometry and texture data. These are called Tables-Of-Contents, TOCs. The TOCs come in shared data and per-instance varieties. The atlas format is a container for these TOCs. The various script commands generate or manipulate these TOCs.
Then you have code to render the geometry, put the texture data into a clipmap and render that, etc. So in concept it is fairly straight forward, but there are a lot of hairy details to it, as well as bit rot that has since crept into the system.
Rene Damm is probably the best expert on all that now. He's pretty active on the forums, hit him up for more information. :)
02/27/2009 (10:20 am)
Bill is right - ChunkedLOD is the inspiration for the geometry. Future tech is based off of Blow's Unified LOD work.Read the SGI Clipmap paper for details on how the clipmap system works. I also wrote an article for Game Programming Gems 7 on the technique.
There are some docs that I wrote explaining how the system works, but I honestly don't know where they would be now - they should be included in the distribution or up on TDN. They covered the script commands, general overview, etc.
The best guide is the source code, at least as far as the internals go. Every class should be documented pretty thoroughly with Doxygen.
Basically you have multiple quadtrees of data, for geometry and texture data. These are called Tables-Of-Contents, TOCs. The TOCs come in shared data and per-instance varieties. The atlas format is a container for these TOCs. The various script commands generate or manipulate these TOCs.
Then you have code to render the geometry, put the texture data into a clipmap and render that, etc. So in concept it is fairly straight forward, but there are a lot of hairy details to it, as well as bit rot that has since crept into the system.
Rene Damm is probably the best expert on all that now. He's pretty active on the forums, hit him up for more information. :)
#14
Awesomeness, thanks!
02/27/2009 (10:58 am)
Hmm. Friday afternoons: the perfect time to blow off 'real' work and read interesting stuff.Awesomeness, thanks!
Torque 3D Owner Ted Southard
In the end, there are enough similarities to do this if you have the skill, and an enterprising community member can make quite a name for themselves by porting a number of resources to the new engines or revisions.
2) That's a good point. Atlas was one of the bigger features of TGEA, and when it came out, it was found to be quite unwieldy for a number of reasons. It's a concern of mine as well, but I don't think that other parts of the engine are in the same boat right now as Atlas is. I can't speak to the future, and you'll probably want to do some searches on this to know more.
3) Check this thread for more information on T3D. It's likely the best source for that info right now. Also, you can check Brett Seyler's (sp?) blogs for some other info and planned eye-candy.
Hope that helps.