How to Create Quality Terrain Textures?
by addiktive · in Torque Game Engine Advanced · 12/22/2006 (6:11 pm) · 19 replies
Hi,
The team and I have been working for a while trying to create suitable quality terrain textures, without massive filesizes. We are generating terrain cell through L3DT and importing them no problem, to ensure we can create high texture quality for each cell, we have split them up into cells. And added the atlas files via the mission file.
Issue 1 is that its pretty much impossible to line the atlas terrain up perfectly.
Issue 2 is that the textures are the worst Ive ever seen... we have a high quality texture artist, but the end result is dismal at best. We cant go any higher quality with the texture, because we will end up with massive file sizes. As it stands each cell is around 70mb...
Why is it that when you generate the atlas file it bloats the texture to such a huge amount?
The overall map size we are creating is roughly 32km x 32km, each cell is around 8kmx8km.
The fact that we cant paint textures, or create holes in atlas terrain is a real downer, we have started expeirmenting with importing a complete Max Model of the terrain and then try to texture that.
Im curious what other devs are doing with large terrain masses?
Are you experiencing massive file sizes and poor texture quality?
Do you have any workarounds, or tips?
This is a previous post we made regarding filesize... but waiting for a repsonse is like waiting for Duke Nukem 3...
I just want to check that the results I'm getting with filesizes are what you would expect.
RAW file, 2048x2048 and a JPG file of 8192x8192.
I then use the following code:
setLogMode(6);
atlasOldGenerateChunkFileFromRaw16("legend/data/terrains/Nuin.raw", 2048, 4, 0.0126, "legend/data/terrains/Nuin.chu", 1.5, 5);
importOldAtlasCHU("legend/data/terrains/Nuin.chu", "legend/data/terrains/Nuin_geometry.atlas");
atlasGenerateTextureTOCFromLargeJPEG("legend/data/terrains/Nuin.jpg", 5, "legend/data/terrains/Nuin_texture.atlas");
atlasGenerateUniqueTerrain("legend/data/terrains/Nuin.atlas", "legend/data/terrains/Nuin_geometry.atlas", "legend/data/terrains/Nuin_texture.atlas");
echo("done");
This produces for me a Nuin.atlas file of 130 meg! 100 meg of which is texture.
Is this correct? There will be 13 "tiles" of 2048x2048, which would mean a total of 1.5-2gig for just landscapes!
The final texture doesn't appear too high res in the engine.
Are there some recommended settings?
Thanks for any help.. If anyone who is creating large terrain in TSE would like to discuss it, please either visit our forum: www.legendrpg.com or email me: addictive@ihug.co.nz
And pray that TSE v1.0 is released sometime this century....?
The team and I have been working for a while trying to create suitable quality terrain textures, without massive filesizes. We are generating terrain cell through L3DT and importing them no problem, to ensure we can create high texture quality for each cell, we have split them up into cells. And added the atlas files via the mission file.
Issue 1 is that its pretty much impossible to line the atlas terrain up perfectly.
Issue 2 is that the textures are the worst Ive ever seen... we have a high quality texture artist, but the end result is dismal at best. We cant go any higher quality with the texture, because we will end up with massive file sizes. As it stands each cell is around 70mb...
Why is it that when you generate the atlas file it bloats the texture to such a huge amount?
The overall map size we are creating is roughly 32km x 32km, each cell is around 8kmx8km.
The fact that we cant paint textures, or create holes in atlas terrain is a real downer, we have started expeirmenting with importing a complete Max Model of the terrain and then try to texture that.
Im curious what other devs are doing with large terrain masses?
Are you experiencing massive file sizes and poor texture quality?
Do you have any workarounds, or tips?
This is a previous post we made regarding filesize... but waiting for a repsonse is like waiting for Duke Nukem 3...
I just want to check that the results I'm getting with filesizes are what you would expect.
RAW file, 2048x2048 and a JPG file of 8192x8192.
I then use the following code:
setLogMode(6);
atlasOldGenerateChunkFileFromRaw16("legend/data/terrains/Nuin.raw", 2048, 4, 0.0126, "legend/data/terrains/Nuin.chu", 1.5, 5);
importOldAtlasCHU("legend/data/terrains/Nuin.chu", "legend/data/terrains/Nuin_geometry.atlas");
atlasGenerateTextureTOCFromLargeJPEG("legend/data/terrains/Nuin.jpg", 5, "legend/data/terrains/Nuin_texture.atlas");
atlasGenerateUniqueTerrain("legend/data/terrains/Nuin.atlas", "legend/data/terrains/Nuin_geometry.atlas", "legend/data/terrains/Nuin_texture.atlas");
echo("done");
This produces for me a Nuin.atlas file of 130 meg! 100 meg of which is texture.
Is this correct? There will be 13 "tiles" of 2048x2048, which would mean a total of 1.5-2gig for just landscapes!
The final texture doesn't appear too high res in the engine.
Are there some recommended settings?
Thanks for any help.. If anyone who is creating large terrain in TSE would like to discuss it, please either visit our forum: www.legendrpg.com or email me: addictive@ihug.co.nz
And pray that TSE v1.0 is released sometime this century....?
About the author
#2
For comparison, in Quake Wars the last numbers I saw, which were for a 16kx16k texture map, were around 512mb for a texture dataset. So having to spend 100mb for 1/4 the texture data is about right, in terms of being competitive with even very high-end million-dollar-a-license technology.
If you want a smaller size on disk, use the blended terrain path - it will reduce your texture file size by around a factor of ten in my experience.
Finally, make sure that you're actually displaying all texture levels. Your tree depths look good. However, you can turn on debug rendering for the terrain texturing by setting $AtlasInstance2::renderDebugTextures=1; in theh console. Each texture detail level will then be represented with a different color pattern, which should help you determine if all the available detail is being rendered or not. The most detailed level is mostly red.
(And, yes, we are working on making this whole process much smoother - it's definitely rougher than it needs to be currently.)
12/23/2006 (11:59 am)
Stephen is pretty much right on. Depending on specifics (you'd want to trace things through a debugger to be 100% certain that you're getting correct behavior; the behavior has changed over the different releases of Atlas), though, you're importing from a large JPEG into PNGs stored in a .atlas file. So PNG does give decent compression but JPEG would be better. You can convert to JPEG using the atlasConvertTextureToc command (if my memory is accurate) and specifying a format.For comparison, in Quake Wars the last numbers I saw, which were for a 16kx16k texture map, were around 512mb for a texture dataset. So having to spend 100mb for 1/4 the texture data is about right, in terms of being competitive with even very high-end million-dollar-a-license technology.
If you want a smaller size on disk, use the blended terrain path - it will reduce your texture file size by around a factor of ten in my experience.
Finally, make sure that you're actually displaying all texture levels. Your tree depths look good. However, you can turn on debug rendering for the terrain texturing by setting $AtlasInstance2::renderDebugTextures=1; in theh console. Each texture detail level will then be represented with a different color pattern, which should help you determine if all the available detail is being rendered or not. The most detailed level is mostly red.
(And, yes, we are working on making this whole process much smoother - it's definitely rougher than it needs to be currently.)
#3
The only issue is that it appears we are stuck with sub-standard texture quality?
Thank you and merry xmas!
addikt
12/23/2006 (1:26 pm)
Wow, I am pleasently surprised to have two Employees Post answers. I appreciate the time you took to respond.. I'll look into the Blended option. At least I know that we arent missing anything crucial, and that the current results we are getting are relatively on par with whats expected.The only issue is that it appears we are stuck with sub-standard texture quality?
Thank you and merry xmas!
addikt
#4
12/23/2006 (8:14 pm)
If you use blended textures then the quality looks a lot better, especially if you are using a lower tile size or higher resolution textures. It's not an option for DEMs and such but for a lot of projects it works great. You might want to bump up the number of tiles though.
#5
The clipmap stuff is also really flexible; as long as you can quickly render arbitrary subrectangles of your texture data, either on CPU for upload or on GPU directly into a texture, it's quite easy to implement new texturing sources. Procedural stuff is just going to be a variation on the existing blender (either on GPU or CPU), and unlike TGE's terrain engine, this is just a matter of subclassing AtlasClipMapImageCache, which is a fairly well-defined interface with 3 implementations in the engine currently, rather than a very cryptic piece of functionality intimately tied to the rest of the rendering system.
12/23/2006 (11:05 pm)
Just for reference: there's no technical reason (although there have occasionally been bugs, hopefully all now fixed) that the blended vs. unique terrains can't look identically good. Obviously it's much easier to create a very high-res blended terrain that might take a gigabyte of unique texture data to match, but that's the whole point of the trade-off! :)The clipmap stuff is also really flexible; as long as you can quickly render arbitrary subrectangles of your texture data, either on CPU for upload or on GPU directly into a texture, it's quite easy to implement new texturing sources. Procedural stuff is just going to be a variation on the existing blender (either on GPU or CPU), and unlike TGE's terrain engine, this is just a matter of subclassing AtlasClipMapImageCache, which is a fairly well-defined interface with 3 implementations in the engine currently, rather than a very cryptic piece of functionality intimately tied to the rest of the rendering system.
#6
I've read http://tdn.garagegames.com/wiki/TSE/Atlas/Using_Atlas2 but it doesn't explain the variables required.
Also, it just shows FromTile commands, while the files I've got are just single files.
I've got the light map, 10 alpha files (one alpha per texture) and the textures.
So I need to know the commands for non-tiled source images, and how to make use of the alpha maps. The rest of it is pretty straightforward.
I assume there is no limit on the number of textures used?
12/24/2006 (3:38 am)
Could someone help explain a bit more the process of using L3DT files to a blended terrain atlas file?I've read http://tdn.garagegames.com/wiki/TSE/Atlas/Using_Atlas2 but it doesn't explain the variables required.
Also, it just shows FromTile commands, while the files I've got are just single files.
I've got the light map, 10 alpha files (one alpha per texture) and the textures.
So I need to know the commands for non-tiled source images, and how to make use of the alpha maps. The rest of it is pretty straightforward.
I assume there is no limit on the number of textures used?
#7
A single file is the same as one tile. :) Just tell it you have 1 tile, and remove the %s stuff from the file string, and it'll work great. You want L3DT to export an alpha map with 4 channels, however.
12/24/2006 (3:20 pm)
It's currently limited to four textures.A single file is the same as one tile. :) Just tell it you have 1 tile, and remove the %s stuff from the file string, and it'll work great. You want L3DT to export an alpha map with 4 channels, however.
#8
Would it be possible if I split a landscape into 3 or 4 atlas files, and put one on top of the other, this will allow more textures in the landscape, based on the atlitude.
So for example, an atlas file of just seabed, then on top of that beach and lowlands, then another of mountains.
It'll require atlas files to be overlapping, will this cause problems?
One benefit is that if the land is mostly grassland, with one "small mountain", then the atlas file for the mountain will be a smaller filesize tile.
Using this method we would be able to eliminate any noticable seams between tiles, since the landscape would be built up vertically using multiple atlas files instead of horizontally using just one atlas file.
edit:
I've tried the blended method, but the process is failing, about a maximum PNG size of 1024.
So we can't do a detailed opacity map for a heightmap of size 2048.
In effect, we would need to do heightmaps in chunks of 512 to get a X2 resolution opacity map?
12/25/2006 (3:02 am)
Thanks for that.Would it be possible if I split a landscape into 3 or 4 atlas files, and put one on top of the other, this will allow more textures in the landscape, based on the atlitude.
So for example, an atlas file of just seabed, then on top of that beach and lowlands, then another of mountains.
It'll require atlas files to be overlapping, will this cause problems?
One benefit is that if the land is mostly grassland, with one "small mountain", then the atlas file for the mountain will be a smaller filesize tile.
Using this method we would be able to eliminate any noticable seams between tiles, since the landscape would be built up vertically using multiple atlas files instead of horizontally using just one atlas file.
edit:
I've tried the blended method, but the process is failing, about a maximum PNG size of 1024.
So we can't do a detailed opacity map for a heightmap of size 2048.
In effect, we would need to do heightmaps in chunks of 512 to get a X2 resolution opacity map?
#9
12/25/2006 (7:29 am)
If you need larger than a 1024x1024 PNG then you could save your opacity image as a PNG mosaic file from L3DT and then use the atlasgenerateatlastexturetocfromtiles to get it imported in.
#10
I've just tried the below, and it seems to process all the way through, but I get a messagebox pop up saying "Failed to Create Bitmap". The final Blended file is being produced, and there are no errors in the log file.
The produced alpha atlas file is 0kb, so it's failing the import, without any errors being reported.
atlasOldGenerateChunkFileFromRaw16("legend/data/terrains/Nuin.raw", 2048, 4, 991.85/65536, "legend/data/terrains/Nuin.chu", 1, 4);
// Import from old CHU format to .atlas.
importOldAtlasCHU("legend/data/terrains/Nuin.chu", "legend/data/terrains/Nuin_geometry.atlas");
// Import data from some tiles we rendered out with L3DT.
atlasGenerateTextureTOCFromTiles(4,"legend/data/terrains/Nuin_Alpha1/Alpha1_x%dy%d.png","legend/data/terrains/Nuin_alpha.atlas", 0);
atlasGenerateTextureTOCFromLargeJPEG("legend/data/terrains/Nuin_LM.jpg", 4, "legend/data/terrains/Nuin_LM.atlas");
atlasGenerateBlenderTerrain(
"legend/data/terrains/NuinBlended.atlas",
"legend/data/terrains/Nuin_Geometry.atlas",
"legend/data/terrains/Nuin_alpha.atlas",
"legend/data/terrains/Nuin_LM.atlas",
// This is the size of the virtual texturemap the blender will be producing. This directly affects how much the source images are tiled, so it is
// hardcoded into the terrain. This must be a power of 2 - if you get it wrong the engine will nicely let you know and suggest some alternatives
32768,
// These are the source textures that the blender will be using. List them in the REVERSE order that L3DT shows in the alpha map export
// wizard. If you specify too few, then your terrain may cause the engine to crash.
"legend/data/terrains/rock1",
"legend/data/terrains/grass1",
"legend/data/terrains/rock1b",
"legend/data/terrains/yellowsand1"
);
// Finally, combine into a unique-textured terrain.
atlasGenerateUniqueTerrain("legend/data/terrains/Nuin.atlas", "legend/data/terrains/Nuin_geometry.atlas", "legend/data/terrains/Nuin_texture.atlas");
12/25/2006 (11:28 am)
Thanks, feel stupid now!I've just tried the below, and it seems to process all the way through, but I get a messagebox pop up saying "Failed to Create Bitmap". The final Blended file is being produced, and there are no errors in the log file.
The produced alpha atlas file is 0kb, so it's failing the import, without any errors being reported.
atlasOldGenerateChunkFileFromRaw16("legend/data/terrains/Nuin.raw", 2048, 4, 991.85/65536, "legend/data/terrains/Nuin.chu", 1, 4);
// Import from old CHU format to .atlas.
importOldAtlasCHU("legend/data/terrains/Nuin.chu", "legend/data/terrains/Nuin_geometry.atlas");
// Import data from some tiles we rendered out with L3DT.
atlasGenerateTextureTOCFromTiles(4,"legend/data/terrains/Nuin_Alpha1/Alpha1_x%dy%d.png","legend/data/terrains/Nuin_alpha.atlas", 0);
atlasGenerateTextureTOCFromLargeJPEG("legend/data/terrains/Nuin_LM.jpg", 4, "legend/data/terrains/Nuin_LM.atlas");
atlasGenerateBlenderTerrain(
"legend/data/terrains/NuinBlended.atlas",
"legend/data/terrains/Nuin_Geometry.atlas",
"legend/data/terrains/Nuin_alpha.atlas",
"legend/data/terrains/Nuin_LM.atlas",
// This is the size of the virtual texturemap the blender will be producing. This directly affects how much the source images are tiled, so it is
// hardcoded into the terrain. This must be a power of 2 - if you get it wrong the engine will nicely let you know and suggest some alternatives
32768,
// These are the source textures that the blender will be using. List them in the REVERSE order that L3DT shows in the alpha map export
// wizard. If you specify too few, then your terrain may cause the engine to crash.
"legend/data/terrains/rock1",
"legend/data/terrains/grass1",
"legend/data/terrains/rock1b",
"legend/data/terrains/yellowsand1"
);
// Finally, combine into a unique-textured terrain.
atlasGenerateUniqueTerrain("legend/data/terrains/Nuin.atlas", "legend/data/terrains/Nuin_geometry.atlas", "legend/data/terrains/Nuin_texture.atlas");
#11
I suspect that you are trying to create the opacity map (a.k.a alpha map) in wrong bitmap format; the output format parameter in your script call is set to 0, which is JPEG, while the alpha/opacity maps require full 32 bits to represent the four 8-bit channels. Therefore you need to use PNG as the output format (value 1):
atlasGenerateTextureTOCFromTiles(4,"legend/data/terrains/Nuin_Alpha1/Alpha1_x%dy%d.png","legend/data/terrains/Nuin_alpha.atlas", 1);
12/25/2006 (2:37 pm)
@addiktiveI suspect that you are trying to create the opacity map (a.k.a alpha map) in wrong bitmap format; the output format parameter in your script call is set to 0, which is JPEG, while the alpha/opacity maps require full 32 bits to represent the four 8-bit channels. Therefore you need to use PNG as the output format (value 1):
atlasGenerateTextureTOCFromTiles(4,"legend/data/terrains/Nuin_Alpha1/Alpha1_x%dy%d.png","legend/data/terrains/Nuin_alpha.atlas", 1);
#12
Produced the atlas file just fine, but the mission will not load!
I get the following error
AtlasChunk: readfromstream - (sent1) invalid Chunk master setinel.
12/25/2006 (3:29 pm)
Thanks, that was it.Produced the atlas file just fine, but the mission will not load!
I get the following error
AtlasChunk: readfromstream - (sent1) invalid Chunk master setinel.
#13
I suggest making sure you can export using a debug build of TSE, and that your exporter and importer are in synch (ie, no old binaries).
12/25/2006 (3:56 pm)
You have a corrupt .atlas file, invalid data in a chunk (maybe you have a chunk that's too big?).I suggest making sure you can export using a debug build of TSE, and that your exporter and importer are in synch (ie, no old binaries).
#14
If you look at this thread (http://www.garagegames.com/mg/forums/result.thread.php?qt=48469)
and scroll down to my reply, I discuss the how and why to have terrain tiles mate together seemlessly.
I hope it was a typo when you said that your heightfields were 2048. It is correct to use the
2048 figure in the atlasOldGenerateChunkFileFromRaw16 function call, but the actual file size should
be 2049 x 2049. In your case, the entire heightfield should be 8193 x 8193, from which you cut 16
tiles, each measuring 2049 x 2049, every 2048 "pixels", so that each tile overlaps it's neighbors
by one row and one column. Then when you place these tile in the world every 8192 meters (2048 * 4)
then the topography should mate together beautifully.
12/26/2006 (9:45 am)
"Issue 1 is that its pretty much impossible to line the atlas terrain up perfectly."If you look at this thread (http://www.garagegames.com/mg/forums/result.thread.php?qt=48469)
and scroll down to my reply, I discuss the how and why to have terrain tiles mate together seemlessly.
I hope it was a typo when you said that your heightfields were 2048. It is correct to use the
2048 figure in the atlasOldGenerateChunkFileFromRaw16 function call, but the actual file size should
be 2049 x 2049. In your case, the entire heightfield should be 8193 x 8193, from which you cut 16
tiles, each measuring 2049 x 2049, every 2048 "pixels", so that each tile overlaps it's neighbors
by one row and one column. Then when you place these tile in the world every 8192 meters (2048 * 4)
then the topography should mate together beautifully.
#15
12/26/2006 (10:50 am)
(You may encounter some small inconsistencies on the edges between blocks, as the LOD won't match up perfectly there, but the evidence suggests that this isn't very noticeable.)
#16
Detail maps, are they implemented in the Alpha?
CHU & TQT method vs single Atlas file method. Which is better? If CHU & TQT, then possible reasons why I can't get it to work.
Falling through the terrain at random locations, bug with alpha, or just us?
12/27/2006 (3:45 pm)
Thanks for the great responses, a couple more questions:Detail maps, are they implemented in the Alpha?
CHU & TQT method vs single Atlas file method. Which is better? If CHU & TQT, then possible reasons why I can't get it to work.
Falling through the terrain at random locations, bug with alpha, or just us?
#17
Only thing to do to debug falling-through bugs is to make a journal of the problem happening and stepping through the code to see what's going wrong.
12/27/2006 (5:27 pm)
CHU & TQT are old file formats, don't use them anymore (except for initial import, as noted in the Atlas2 docs). All new Atlas code uses .atlas files and only .atlas files.Only thing to do to debug falling-through bugs is to make a journal of the problem happening and stepping through the code to see what's going wrong.
#18
12/28/2006 (1:28 am)
@addiktive: Detail maps don't work currently in atlas. A few of us have been trying for a while to get them working, there is a hole for them in the code, but can't figure out how to get them to actually render. I've tried doing it through code and rigging it though shaders (bumping up to ps3.0 in order to do so) and been unable to get them going either way.
#19
We will not worry about detail maps at the moment. They are not "important" at the moment since the project is in early landscape stages, so hopefully some time in the future this feature will become available i further updates.
Thanks Mary for the post.
Yes, we are using a 2049x2049 RAW file for a 2048x2048 tile. We are getting a small difference in altitude at the seams, but after reading the topic you posted, I have a couple of ideas about how to fix that.
12/28/2006 (2:00 pm)
Thanks for the replies.We will not worry about detail maps at the moment. They are not "important" at the moment since the project is in early landscape stages, so hopefully some time in the future this feature will become available i further updates.
Thanks Mary for the post.
Yes, we are using a 2049x2049 RAW file for a 2048x2048 tile. We are getting a small difference in altitude at the seams, but after reading the topic you posted, I have a couple of ideas about how to fix that.
Torque 3D Owner Stephen Zepp
Do the math for a moment: You are using an 8192 x 8192 jpg texture. By itself, at one byte of color information, that's worst case 67 meg of texture information right there. Add to that the fact that we usually use 3 bytes of color information (24 bit) for jpg iirc, and your initial textures (again worst case, because jpg does do some lossfull compression), and your initial texture itself is much much bigger than 100 meg.
Added to this, Atlast trades datafile size and pre-processing time for runtime speed, meaning that it packs a lot of information for each of the levels of detail into the pre-processed Atlas files so that the calculations are all done prior to running the game.
Also, you are using an 8192 x 8192 texture to cover 2048 x 2048--that's a lot of texture information for terrain blocks that size.
Finally, jpg's are a lossy texture type. All the processing and also editing (if you perform it) on your textures are causing the compression algorithm to lose information over time, and lowers the quality of the texture itself. In addition, unique terrains aren't designed to be super high quality, but to represent satellite imagery of real world areas, etc....you may want to try using the blended texture capabilities instead.