Terrain Textures/LOD Issues
by Ryan Dey (not Tom Watson) · in Torque Game Engine Advanced · 08/08/2006 (2:29 pm) · 48 replies
I'm nearing completion of my Atlas terrain project, but have one last issue that MUST be dealt with, since its a still deal-breaker at the moment. I have loaded 28 unique terrain tiles, each 1024x1024 meters, with 1-meter horizontal terrain resolution, and 0.25-meter texture resolution. It covers 29.4 square kilometers of land, and the atlas data files total around 1.1 gigs :)
The first issue, seen here:
I am getting a random arrangement of areas where the texture doesn't load, and instead shows white boxes. In other areas, it loads a texture, but not the correct one. It will instead place a texture from some other area in my dataset. This has been occuring from the beginning (when I had 4 tiles), and changes locations every time I load the mission. I have a suspicion its related to TSE not being able to push enough texture data to my video card (which is pretty low end), but I saw similar results on my boss's Alienware Laptop (which can run Oblivion easily). If it is a memory push issue, I'm not sure what I could do to optimize the textures and/or quad-tree. Currently, I'm using source jpgs that are 4096x4096 and leafs that are 256x256 (using a 5 in the creation command).
The second issue, seen here:
The left image is the source jpg, with 9-inch resoltion (scaled down for posting, of course), the right is TSE's display of it, with much much worse resolution . No matter how close I move the camera to the terrain, it does not display the highest resolution imagery, possibly not loading the best possible Levels of Detail (LOD) available.
I am severly hoping this isn't related to tree depths not matching in the elevation and textures. My texture tree depth is set to 5 to create small pieces my video card can handle, and my terrain tree depth is set to 2 because I have a nearly (but not quite) flat study area with very low verts per chunk and a near-zero error metric. Increase terrain tree-depth or decreasing texture tree-depth seems like it would make data streaming less efficient.
Here are the commands I use to create an example atlas tile:
//G06: (1062 verts/chunk)
atlasOldGenerateChunkFileFromRaw16("./elev_G06_1025.raw", 1024, 1.0, 1.0 / 100.0,"./elev_G06_1025.chu", 0.2, 2 );
ImportOldAtlasCHU("./elev_G06_1025.chu","./elev_G06_1025.atlas");
atlasGenerateTextureTOCFromLargeJPEG("./Aerial_G06_4096.jpg", 5, "./aerial_G06_4096.atlas");
atlasGenerateUniqueTerrain("./Data_NDTIP_G06.atlas","./elev_G06_1025.atlas","./aerial_G06_4096.atlas");
Thoughts?
Edit: I needed to figure out how to embed images.
The first issue, seen here:
I am getting a random arrangement of areas where the texture doesn't load, and instead shows white boxes. In other areas, it loads a texture, but not the correct one. It will instead place a texture from some other area in my dataset. This has been occuring from the beginning (when I had 4 tiles), and changes locations every time I load the mission. I have a suspicion its related to TSE not being able to push enough texture data to my video card (which is pretty low end), but I saw similar results on my boss's Alienware Laptop (which can run Oblivion easily). If it is a memory push issue, I'm not sure what I could do to optimize the textures and/or quad-tree. Currently, I'm using source jpgs that are 4096x4096 and leafs that are 256x256 (using a 5 in the creation command).The second issue, seen here:
The left image is the source jpg, with 9-inch resoltion (scaled down for posting, of course), the right is TSE's display of it, with much much worse resolution . No matter how close I move the camera to the terrain, it does not display the highest resolution imagery, possibly not loading the best possible Levels of Detail (LOD) available.I am severly hoping this isn't related to tree depths not matching in the elevation and textures. My texture tree depth is set to 5 to create small pieces my video card can handle, and my terrain tree depth is set to 2 because I have a nearly (but not quite) flat study area with very low verts per chunk and a near-zero error metric. Increase terrain tree-depth or decreasing texture tree-depth seems like it would make data streaming less efficient.
Here are the commands I use to create an example atlas tile:
//G06: (1062 verts/chunk)
atlasOldGenerateChunkFileFromRaw16("./elev_G06_1025.raw", 1024, 1.0, 1.0 / 100.0,"./elev_G06_1025.chu", 0.2, 2 );
ImportOldAtlasCHU("./elev_G06_1025.chu","./elev_G06_1025.atlas");
atlasGenerateTextureTOCFromLargeJPEG("./Aerial_G06_4096.jpg", 5, "./aerial_G06_4096.atlas");
atlasGenerateUniqueTerrain("./Data_NDTIP_G06.atlas","./elev_G06_1025.atlas","./aerial_G06_4096.atlas");
Thoughts?
Edit: I needed to figure out how to embed images.
About the author
#2
Please do!
08/08/2006 (5:06 pm)
Quote:
If you have extremely large datasets, there are many things that can be done to get them succesfully processed and imported - if you want I can outline some.
Please do!
#3
For heightfields over, say, 8k square, you may have to write a tool to take several geometry TOCs and combine them into a larger one (ie, generating a piece of geometry for the highest level of the quadtree, then storing the geometry from each file into the approriate subtree).
In general here at GG we're not really doing a lot of dev/testing with extremely large datasets, so if you try to push the tech beyond what can be fit into a single source file, you might have to extend it yourself to get where you need to go. The basic architecture is extremely scalable - just some of the associated tools that tend to break in those situations.
08/08/2006 (5:10 pm)
Well, as far as texture import goes, you just have to have a big enough jpeg or tile set. No magic there.For heightfields over, say, 8k square, you may have to write a tool to take several geometry TOCs and combine them into a larger one (ie, generating a piece of geometry for the highest level of the quadtree, then storing the geometry from each file into the approriate subtree).
In general here at GG we're not really doing a lot of dev/testing with extremely large datasets, so if you try to push the tech beyond what can be fit into a single source file, you might have to extend it yourself to get where you need to go. The basic architecture is extremely scalable - just some of the associated tools that tend to break in those situations.
#4
The concept of multiple Atlas terrains in one mission is actually an attractive one. My own experience is that even a small 20x20km (that's sort of small for me) terrain is painful to create. It's an all or nothing sort of experience.
Something I'd love to see is the ability to swap in/out complete Atlas instances. One could have a mission file that defines game space in multiple Atlas instances, then allow the client to only load them up as needed based on maximum visibility.
For my own project I'm aiming to map an area on the order of 100x50 km minimum. Ideally what I'd like to do is create say 200 5x5km atlas instances and then tile them together. Using that set I could create simple small missions or put them all in one large mission. I also could create varying detail levels - for example maybe replacing one "low res" instance with a new "high res" instance if I acquire new finer scale data. From the game development point of view this also would make sense I think. I might create a finer scale atlas terrain for an area where I had detailed gameplay (like in a town or village), and leave the surrounding land relatively low resolution.
Also if I did have a system where I could "swap" atlas instances, I'd probably want to parent other items to the Atlas instance. So for example foliage would swap in and out with the Atlas instance it was parented to.
08/08/2006 (5:38 pm)
/me puts on the "brainstorm" hat...The concept of multiple Atlas terrains in one mission is actually an attractive one. My own experience is that even a small 20x20km (that's sort of small for me) terrain is painful to create. It's an all or nothing sort of experience.
Something I'd love to see is the ability to swap in/out complete Atlas instances. One could have a mission file that defines game space in multiple Atlas instances, then allow the client to only load them up as needed based on maximum visibility.
For my own project I'm aiming to map an area on the order of 100x50 km minimum. Ideally what I'd like to do is create say 200 5x5km atlas instances and then tile them together. Using that set I could create simple small missions or put them all in one large mission. I also could create varying detail levels - for example maybe replacing one "low res" instance with a new "high res" instance if I acquire new finer scale data. From the game development point of view this also would make sense I think. I might create a finer scale atlas terrain for an area where I had detailed gameplay (like in a town or village), and leave the surrounding land relatively low resolution.
Also if I did have a system where I could "swap" atlas instances, I'd probably want to parent other items to the Atlas instance. So for example foliage would swap in and out with the Atlas instance it was parented to.
#5
The grid approach could be good but you'd want to encapsulate it in an AtlasGrid or something like that - otherwise you're going to have LOD issues, paging performance issues, etc. Doing that approach would also let you avoid the memory hits, have a better framework for incremental paging of game objects, etc.
I have some improvements in mind for enhancing support for, say, per instance lightmaps, but they won't (can't?) address the major issue you're hitting now.
08/08/2006 (11:10 pm)
BTW - what are your resolutions in terms of heightfield/texture size? Insofar as Atlas is concerned that's the main factor. How that data is scaled once it's in-game is not so important (within limits).The grid approach could be good but you'd want to encapsulate it in an AtlasGrid or something like that - otherwise you're going to have LOD issues, paging performance issues, etc. Doing that approach would also let you avoid the memory hits, have a better framework for incremental paging of game objects, etc.
I have some improvements in mind for enhancing support for, say, per instance lightmaps, but they won't (can't?) address the major issue you're hitting now.
#6
08/09/2006 (2:57 am)
Interesting discussion. Those images above look like "screenshots" from Google Earth... :-)
#7
In terms of making it all a single .atlas file, I don't really believe that to be an option for us at this point. One big elevation geometry I could do (it would be a huge file at the 1-meter resolution I'm using, but could tone it down to as low as 10-meter resolution). The imagery, however, just won't work at that size. The square extent of my study area is 16kmx16km, and I need the very highest resolution I can get. Our source imagery is in the 5.5 to 6 inch range (better than Google Earth, and not freely available), but was settling for the 9 inch (0.25m) I could get using the 4096 pixel size. At 16km square, we'd need a jpg that's 65k pixels on a side, which is literally impossible. I remember reading atlas can work with jpgs up to 8192 pixels, so the very biggest tiles we could use at the image resolution we need is 2kmx2km.
Essentially, Atlas can support more data than the size of the universe, but if you want to work with unique textures (such as any real-world data), you are limited to 8192 times the resolution of your imagery. This is impracticle for our needs, and possibly most others doing visualization work. We will have to come up with some way to work around this issue or we can't go on spending resources on TSE.
I remember in the game Black and White, you could start by looking at an entire island, and continue zooming in until you were looking at a worm in an apple. In Oblivion, you can stand on a mountain on one side of the map, look across the world and see a mountain 10's of kilometers away and then walk to it. In MS FLight Simulator, you can fly a Cessna from Eugene Oregon to Lewes Delaware and see 15-meter terrain or better the whole way (with procedural texturing). These examples use levels of detail to only load the needed information to the video card. Can TSE do this or not? (I know it could be extended to do dern near anything, but I have a LONG way to go before I know enough to re-program the engine).
Ok... To the second issue:
So I thought I'd conclude with a brief discussion of the resolutions of sources, and what I'm loading into TSE.
Study Area: We are working with a proposed tranportation feature (road, associated bridges, etc), that is about 15 kilometers long, but since its oriented NW/SE, it fits within a 10kmx13km box. In order to keep things square (power of 2) and provide some extra edges space, I've defined a 16km x 16km study area (which I then cut into 256 tiles 1024 meters on a side). Each of these tiles is being converted to individual atlas instances.
Elevation data: I am currently working with the USGS 10-m (horizontal) dataset because it covers the entire area were are working with. We were considering adding LIDAR (sub-inch) data to the areas where we have that, but that hasn't happened yet. I am exporting that to raw files with 1-meter horizontal resolution because that makes the math easy, allows for a good amount of overlap between tiles, and was quite convenient when I was using Terragen for exporting raws. I have considered lowering the resolution (which would allow larger atlas tiles), but that wouldn't help the verts/chunk issue, and geometry is such a small proportion of the final data file sizes that it doesn't seem worth it.
Texture data: We have 6-inch (0.15meter) aerials that cover the length of our study area. I have been downsampling it to 0.25m horizontal resolution, giving it 4096 pixels when cut to a tile extent. In our visualization, the resolution of the aerial of of critical importance, and we need as much resolution as we can get (which restrains us from creating huge atlas tiles, considering a single jpg has to be stretched across an entire atlas instance).
You mentioned that you don't test your engine with large datasets, and I can understand. To me (and Tim, and others) games are always pushing the bounds of technology advancement, making them an ideal platform to do "scientific" work work such as non-game visualizations. I'm happy to push your technology further than you inteded to go, and understand that breaking it is just part of the process. It's fun to load 30 square kilometers of terrain data into an engine designed for 4.
08/09/2006 (11:02 am)
Good morning, and thanks for getting the discussion rolling while I was sleeping... helps me jump right in this morning :)Quote:I admittedly don't understand the .atlas file structure yet, but I had assumed that all the important "source" information was contained in each one. Are you referring to the .atlas containing geometry and textures as the source? The other part I don't understand is the paging system (I'm new to this, need more time to dig). I understand that each atlas instance in my mission needs some resources to track it, but I was under the assumption that if a tile/instance is very vary away from the viewer, only the lowest detail geometry/texture were loaded into the video card. That way, every tile would have some small chunk of memory, and only the closer tiles would have a lot of memory. Is this not the case with the paging system? Maybe I'll look for some documentation on the atlas TOC's and paging stuff.
First off - multiple atlas tiles from different source files is a VERY bad idea in the current implementation. Each seperate source file that is loaded will spawn off a set of loading threads, track resources seperately, etc.
In terms of making it all a single .atlas file, I don't really believe that to be an option for us at this point. One big elevation geometry I could do (it would be a huge file at the 1-meter resolution I'm using, but could tone it down to as low as 10-meter resolution). The imagery, however, just won't work at that size. The square extent of my study area is 16kmx16km, and I need the very highest resolution I can get. Our source imagery is in the 5.5 to 6 inch range (better than Google Earth, and not freely available), but was settling for the 9 inch (0.25m) I could get using the 4096 pixel size. At 16km square, we'd need a jpg that's 65k pixels on a side, which is literally impossible. I remember reading atlas can work with jpgs up to 8192 pixels, so the very biggest tiles we could use at the image resolution we need is 2kmx2km.
Essentially, Atlas can support more data than the size of the universe, but if you want to work with unique textures (such as any real-world data), you are limited to 8192 times the resolution of your imagery. This is impracticle for our needs, and possibly most others doing visualization work. We will have to come up with some way to work around this issue or we can't go on spending resources on TSE.
I remember in the game Black and White, you could start by looking at an entire island, and continue zooming in until you were looking at a worm in an apple. In Oblivion, you can stand on a mountain on one side of the map, look across the world and see a mountain 10's of kilometers away and then walk to it. In MS FLight Simulator, you can fly a Cessna from Eugene Oregon to Lewes Delaware and see 15-meter terrain or better the whole way (with procedural texturing). These examples use levels of detail to only load the needed information to the video card. Can TSE do this or not? (I know it could be extended to do dern near anything, but I have a LONG way to go before I know enough to re-program the engine).
Ok... To the second issue:
Quote:I sorta feared this was the problem. I actually understand why is this the case from my previous experience with computer graphics programming (years ago), and working with Visual Nature Studio (non-real-time visualization). This might be the only area of the engine where I have a good understanding of how TSE works. Unfortunately, making the texture and geometry tree-depths the same is going to cause the compromises I discussed previously. Either I'm going to need less texture leaves (each 1024px instead of 256px) which will be harder to load in to the video card, or I will need to create smaller terrain chunks (with verts/chunk getting extremely low... they currently range from 66 to 2500, and would drop to around 16-500), which I understand is inefficient. At any rate, I'll see if I can make this part work better.
Secondly, in the currently released version of Atlas, only texture data up to the depth of the geometry tree can be rendered.
So I thought I'd conclude with a brief discussion of the resolutions of sources, and what I'm loading into TSE.
Study Area: We are working with a proposed tranportation feature (road, associated bridges, etc), that is about 15 kilometers long, but since its oriented NW/SE, it fits within a 10kmx13km box. In order to keep things square (power of 2) and provide some extra edges space, I've defined a 16km x 16km study area (which I then cut into 256 tiles 1024 meters on a side). Each of these tiles is being converted to individual atlas instances.
Elevation data: I am currently working with the USGS 10-m (horizontal) dataset because it covers the entire area were are working with. We were considering adding LIDAR (sub-inch) data to the areas where we have that, but that hasn't happened yet. I am exporting that to raw files with 1-meter horizontal resolution because that makes the math easy, allows for a good amount of overlap between tiles, and was quite convenient when I was using Terragen for exporting raws. I have considered lowering the resolution (which would allow larger atlas tiles), but that wouldn't help the verts/chunk issue, and geometry is such a small proportion of the final data file sizes that it doesn't seem worth it.
Texture data: We have 6-inch (0.15meter) aerials that cover the length of our study area. I have been downsampling it to 0.25m horizontal resolution, giving it 4096 pixels when cut to a tile extent. In our visualization, the resolution of the aerial of of critical importance, and we need as much resolution as we can get (which restrains us from creating huge atlas tiles, considering a single jpg has to be stretched across an entire atlas instance).
You mentioned that you don't test your engine with large datasets, and I can understand. To me (and Tim, and others) games are always pushing the bounds of technology advancement, making them an ideal platform to do "scientific" work work such as non-game visualizations. I'm happy to push your technology further than you inteded to go, and understand that breaking it is just part of the process. It's fun to load 30 square kilometers of terrain data into an engine designed for 4.
#8
Or even 400 :^)
My data is 10 meter DEM data which I'm overlaying 1/2 meter resolution orthoquads. I'm also working with a 30 meter resolution forest composition dataset. The aim is something like this...
08/09/2006 (11:26 am)
Quote:
It's fun to load 30 square kilometers of terrain data into an engine designed for 4.
Or even 400 :^)
My data is 10 meter DEM data which I'm overlaying 1/2 meter resolution orthoquads. I'm also working with a 30 meter resolution forest composition dataset. The aim is something like this...
#9
We are looking to produce real-time versions of these:
Although we, of course, understand that real-time tools (like TSE) aren't ready to handle all that processing yet. But we can try.
08/09/2006 (11:40 am)
Heh... that looks almost like what I was thinking about in Oblivion.We are looking to produce real-time versions of these:
Although we, of course, understand that real-time tools (like TSE) aren't ready to handle all that processing yet. But we can try.
#11
08/09/2006 (11:46 am)
Nice eye-candy :-)
#12
Unfortunately no control over individual tree placement (yet). Their methods of actually placing terrain in-game are a bit easier to do, plus the upcoming technology that allows creating specific overlayed areas of finer detail terrain is also a win, along with integration of SpeedTree.
08/09/2006 (12:09 pm)
@ CliffQuote:Tim - Not liking Multiverse's terrain engine? ;-)
Unfortunately no control over individual tree placement (yet). Their methods of actually placing terrain in-game are a bit easier to do, plus the upcoming technology that allows creating specific overlayed areas of finer detail terrain is also a win, along with integration of SpeedTree.
#13
Cool goals! I think at least visually they're within reach of modern hardware.
(I would like to mention that if you need extremely large dataset support, our support guy, Stephen Zepp, has been working with several groups on similar issues, and may be able to help you, too. Just toss him an e-mail at stephenz@garagegames.com if you think a support contract might be something you need. Meanwhile, I'm happy to discuss things here. :)
@Ryan:
Actually, a 64k px JPEG is perfect - that's the max size that JPEG supports, and the import tools for Atlas should deal with it just fine. If they don't, let me know - it's probably a bug that should get fixed. Understandably, we don't do heavy testing on the extremes, but I would like to make that situation work for people.
The _hardware_ only supports a tile size of between 1k to 4k depending on card (all current gen cards support 4k px textures), but Atlas is designed to virtualize all that away... :)
As for tree depth - if you HAVE to have subdivision, you gotta have it and you may as well take the hit. :) It's not the end of the world, just is a little additional hit to the renderer. On the todo list is some stuff to allow additional subdivision of chunks w/o having to have deeper trees; if someone wants to help with it it's a nice self-contained task, probably a day or two of work, and I'd be willing to pay a bounty for it!
In the next push of the Atlas tech, I've implemented a clipmap solution that frees you to some extent from having tree depth constraints. As long as they're in the same rough range you'll be ok. Right now we're doing testing/performance tuning on it, and getting it as solid as possible before release.
Atlas also supports importing discrete tiles, so if you have data larger than 64k px on a size (or already have it in tiles) you can save it out as a bunch of PNG or JPEG files, and import it that way. Using this import path lets you effectively work with any size image dataset. The only real constraint here is that you may have some issues with very large file sizes - I think right now > 2gb will give you some problems; if you run into that limit I have a patch to get you up to 4gb. Beyond that, you may have to do some tinkering with Torque's file IO code to make sure it's 64bit all the way through, but it should be relatively easy to get up to 16gb or larger. Your OS may become an issue at this point.
For the heightfields, Atlas is known to work up with heightfields that are 8192x8192. For your dataset, that sounds like it would let you have 2m grid density, which seems reasonable. A better mesh generation path is also in progress that will scale to even larger datasets; that may not be in the next release, unfortunately.
And you could always do the quadtree combination trick I mentioned, in which case you could import your heightfield as tiles, and then just have a bit of weirdness in the lowest detail level (which you'd likely never see).
TSE definitely supports powerful LOD - you just have to learn how to work with it. ;)
08/09/2006 (2:30 pm)
Hey guys,Cool goals! I think at least visually they're within reach of modern hardware.
(I would like to mention that if you need extremely large dataset support, our support guy, Stephen Zepp, has been working with several groups on similar issues, and may be able to help you, too. Just toss him an e-mail at stephenz@garagegames.com if you think a support contract might be something you need. Meanwhile, I'm happy to discuss things here. :)
@Ryan:
Actually, a 64k px JPEG is perfect - that's the max size that JPEG supports, and the import tools for Atlas should deal with it just fine. If they don't, let me know - it's probably a bug that should get fixed. Understandably, we don't do heavy testing on the extremes, but I would like to make that situation work for people.
The _hardware_ only supports a tile size of between 1k to 4k depending on card (all current gen cards support 4k px textures), but Atlas is designed to virtualize all that away... :)
As for tree depth - if you HAVE to have subdivision, you gotta have it and you may as well take the hit. :) It's not the end of the world, just is a little additional hit to the renderer. On the todo list is some stuff to allow additional subdivision of chunks w/o having to have deeper trees; if someone wants to help with it it's a nice self-contained task, probably a day or two of work, and I'd be willing to pay a bounty for it!
In the next push of the Atlas tech, I've implemented a clipmap solution that frees you to some extent from having tree depth constraints. As long as they're in the same rough range you'll be ok. Right now we're doing testing/performance tuning on it, and getting it as solid as possible before release.
Atlas also supports importing discrete tiles, so if you have data larger than 64k px on a size (or already have it in tiles) you can save it out as a bunch of PNG or JPEG files, and import it that way. Using this import path lets you effectively work with any size image dataset. The only real constraint here is that you may have some issues with very large file sizes - I think right now > 2gb will give you some problems; if you run into that limit I have a patch to get you up to 4gb. Beyond that, you may have to do some tinkering with Torque's file IO code to make sure it's 64bit all the way through, but it should be relatively easy to get up to 16gb or larger. Your OS may become an issue at this point.
For the heightfields, Atlas is known to work up with heightfields that are 8192x8192. For your dataset, that sounds like it would let you have 2m grid density, which seems reasonable. A better mesh generation path is also in progress that will scale to even larger datasets; that may not be in the next release, unfortunately.
And you could always do the quadtree combination trick I mentioned, in which case you could import your heightfield as tiles, and then just have a bit of weirdness in the lowest detail level (which you'd likely never see).
TSE definitely supports powerful LOD - you just have to learn how to work with it. ;)
#14
Thanks for the reply. Some of the stuff (discrete tiles, then importing, etc) seems a bit over my head at the moment, but I'm sure I'll reread it a few times and start to figure it all out.
I may try to make a jpg that's 65k pixels on a side, but I shudder to think what it's going to do to my hard drive. The source imagery is around 2.8 gigs, broken into 50 sections, so even a downsample is going to be a monster as a single file. I don't think Photoshop could open it (30k limit?) so I'm not sure how I will do the mirror and channel swapping currently needed to get it high,wide, and handsome.
One of the things I liked about the small 1km tiles was that I wouldn't have to load anything that's far away from the study area. With larger tiles, I'll have to load in much more data than is needed. Not a terrible thing, but we may have to think about fitting in on a DVD instead of a pair of CD's.
What I may do, actually, as an iterative testing development process, is create 4 tiles that are 8km on a side, with slightly more managable jpgs (32k square). Or, since I don't really even need the southern 4km of the study area, find some creative way to tile 8k and 4k squares. I would still have multiple tiles, but it just might work better.
BTW, I already have the geometry raw file that's 65536 pixels wide. I've been using it to cut out the individual 1km tiles. It's only 600 megs :)
08/09/2006 (3:56 pm)
Hey Ben,Thanks for the reply. Some of the stuff (discrete tiles, then importing, etc) seems a bit over my head at the moment, but I'm sure I'll reread it a few times and start to figure it all out.
I may try to make a jpg that's 65k pixels on a side, but I shudder to think what it's going to do to my hard drive. The source imagery is around 2.8 gigs, broken into 50 sections, so even a downsample is going to be a monster as a single file. I don't think Photoshop could open it (30k limit?) so I'm not sure how I will do the mirror and channel swapping currently needed to get it high,wide, and handsome.
One of the things I liked about the small 1km tiles was that I wouldn't have to load anything that's far away from the study area. With larger tiles, I'll have to load in much more data than is needed. Not a terrible thing, but we may have to think about fitting in on a DVD instead of a pair of CD's.
What I may do, actually, as an iterative testing development process, is create 4 tiles that are 8km on a side, with slightly more managable jpgs (32k square). Or, since I don't really even need the southern 4km of the study area, find some creative way to tile 8k and 4k squares. I would still have multiple tiles, but it just might work better.
BTW, I already have the geometry raw file that's 65536 pixels wide. I've been using it to cut out the individual 1km tiles. It's only 600 megs :)
#15
08/09/2006 (4:10 pm)
If the texture data is already in tiles, why not just load those with the tile import command?
#16
08/09/2006 (4:13 pm)
Hmm... good question. I'm not familar with a tile import command. I'll take a look through the docs tomorrow when I have time to play with TSE again. (At the moment I'm making maps for ODOT).
#17
Lives in atlasImportTile.cpp
See the atlas2 TDN page for an example of use. I have more docs en route that will launch with next TSE push.
08/09/2006 (6:11 pm)
ConsoleFunction(atlasGenerateTextureTOCFromTiles, void, 5, 5, "(leafCount, tileMask, outFile, outFormat) - "
"Generate a texture TOC from a set of tiles. leafCount is the size of the grid "
"of tiles on a side. tileMask is the path for the tiles (no extension) with %d "
"for x and y, ie, 'demo/alpha/Alpha1_x%dy%d'. outFile is the file to output a "
"new .atlas file to.")Lives in atlasImportTile.cpp
See the atlas2 TDN page for an example of use. I have more docs en route that will launch with next TSE push.
#18
Edit: The matching tree-depth thing does seem to mostly fix the blurry texture issue. I think the docs should mention how important that aspect is (even if its being mitigated in the next code drop).
08/10/2006 (8:27 am)
Hey, that looks useful. Handy thing to have in the engine. I'll check it out today and see if I can make some magic.Edit: The matching tree-depth thing does seem to mostly fix the blurry texture issue. I think the docs should mention how important that aspect is (even if its being mitigated in the next code drop).
#19
I'm looking forward to the new code release. Let us all know when its ready!
08/10/2006 (8:55 am)
That function was bugged last time I tried it, Ben. It was giving me a "cannot log PNG" error when I ran it. Though, I might have been running it from the console (after the engine starts) and I know there was some problems with that in the last release so maybe I'll give it another try running from main.cs.I'm looking forward to the new code release. Let us all know when its ready!
#20
I don't have the source code here to look at to see what I'm doing wrong, but my guess is I'm using the function poorly.... It's not entirely clear how it should be used.
Here's what I'm doing:
I have 4 PNG texture files, which I want to tile in a 2x2 grid.
I've called them Aerial_X1Y1.png, Aerial_X1Y2.png, Aerial_X2Y1.png, and AerialX2Y2.png
I call the tile function (from the engine-loaded console) as
atlasGenerateTextureTOCFromTiles(2, "./aerial_X%dY%d", "./aerial_E08F09.atlas");
I have no idea how this new texture atlas would have a tree depth specified, so I tried it wil .atlas files as well and got identical results. I also don't have enough TSE competancy yet to understand the ConsoleFunction example given above, so I couldn't get that working either (and it has an extra parameter at the end).
So, I really don't have time to play with tiles at the moment (though they do sound like I could use them to load managable textures into a giant atlas covering the 16 kilometer square area). For the time being, I'm going to make 4kmx4km atlas instances and see how well they work, then move to 8x8 (of which I'd need only 4).
08/10/2006 (10:01 am)
Yah, I'm getting a fatal error in atlasTexChunk.cpp as follows: "AtlasTexChunk::write - (PNG) no bitmap is present!"I don't have the source code here to look at to see what I'm doing wrong, but my guess is I'm using the function poorly.... It's not entirely clear how it should be used.
Here's what I'm doing:
I have 4 PNG texture files, which I want to tile in a 2x2 grid.
I've called them Aerial_X1Y1.png, Aerial_X1Y2.png, Aerial_X2Y1.png, and AerialX2Y2.png
I call the tile function (from the engine-loaded console) as
atlasGenerateTextureTOCFromTiles(2, "./aerial_X%dY%d", "./aerial_E08F09.atlas");
I have no idea how this new texture atlas would have a tree depth specified, so I tried it wil .atlas files as well and got identical results. I also don't have enough TSE competancy yet to understand the ConsoleFunction example given above, so I couldn't get that working either (and it has an extra parameter at the end).
So, I really don't have time to play with tiles at the moment (though they do sound like I could use them to load managable textures into a giant atlas covering the 16 kilometer square area). For the time being, I'm going to make 4kmx4km atlas instances and see how well they work, then move to 8x8 (of which I'd need only 4).
Associate Kyle Carter
First off - multiple atlas tiles from different source files is a VERY bad idea in the current implementation. Each seperate source file that is loaded will spawn off a set of loading threads, track resources seperately, etc.
So your first problem may be related to that.
Secondly, in the currently released version of Atlas, only texture data up to the depth of the geometry tree can be rendered. There's just no way to render texture data any more finely than that in a tile based approach (without very costly rendering tricks) - you can only set one texture at a time for a given chunk of geometry!
The newest version of Atlas, in testing/polish phase now, addresses this to some extent - but even with it for every atlas instance you have rendering you'll be taking an 8 to 32mb memory hit. So in your scene you'd be burning through 224 to 896mb of video ram, just for textures - the current system is a bit better in the best case but it can be much worse in the worst case, and it has a much higher probability of memory fragmentation.
The take away here, for getting your stuff working in the version of Atlas you have now is:
1. Only use one atlas file if you can help it. Especially in this case where you basically have one big terrain, it's not a smart move to break things up if you can help it.
2. Make sure geometry tree depth is deep as the texture tree depth, if you want to see all your detail. :)
In the next release the second point won't be so important, but #1 will still be good to keep in mind.
If you have extremely large datasets, there are many things that can be done to get them succesfully processed and imported - if you want I can outline some.