Normal Map Formats
by Timmy01 · 07/05/2014 (7:19 pm) · 26 comments
Currently we have a few different options to supply a normal map. Let's check out the quality first and than I'll explain the formats.
PNG
DXT1
DXT3
DXT5nm
DXT5_xGxR
DXT5_oGxR
3Dc
Before we get into any explanations, I'm sure everyone has noticed what a horrible format DXT1/DXT3 is for normal maps. I won't be discussing these two formats.
DXT5
This format can be a cause of confusion in T3D. T3D currently treats any DXT5 texture that is marked for use as a normal map as a special DXT5nm format. If you don't supply this texture in the correct format you will get very strange results indeed. This format differs from the rest by taking advantage of a great trick available with the DXT5 format. The DXT5 format has 4 channels and is laid out like so: Red 5bits | Green 6bits | Blue 5bits | Alpha 8bits.
So what we do here is take advantage of the fact that the alpha and the green channel have the highest bit count. We dump the blue (or Z) channel and we can re-create this value in our pixel shader, great now we are left with only two channels we need to worry about. So we take advantage of that alpha channel and move the red channel into it and leave the green channel as is (remember she has 6bits). Now in the pixel shader we have to do a swizzle trick that reads in the alpha and green channel.
Now to get the best rest results when using this nifty DXT5 trick, we really need to fill the vacant Red and Blue channel with the same solid color (DXT5_xGxR above). One thing that the NVidia DDS photoshop plugin does by default, is when you select DXT5NM it leaves the red/blue channel as is (DXT5NM above). This produces results that are not as good because of the way DXT5 is compressed. The other format is DXT5_oGxR, currently T3D has a class called DXT5nmSwizzle that when used it writes 0's to the red channel and 1's to the blue channel. Again this doesn't allow the best quality available but is better than the default photoshop way (actually this is the format if you happened to use my PR github.com/GarageGames/Torque3D/pull/715 for enabling parallax support with DXT5).
3Dc/BC5
Now finally we have the awesome 3Dc (aka Block Compression 5 (BC5) in Direct3D10 or later). The results are far better than any of the other compressed formats. I have added support for this in my own local repo just for testing it out, there are a few problems associated with adding this to T3D but that is a discussion for another day. This format also uses an almost identical shader trick as DXT5 does.
You can find the original high resolution screenshots here narivtech.com/downloads/results.zip and the normal maps here narivtech.com/downloads/normal_maps.zip
PNG
DXT1
DXT3
DXT5nm
DXT5_xGxR
DXT5_oGxR
3Dc
Before we get into any explanations, I'm sure everyone has noticed what a horrible format DXT1/DXT3 is for normal maps. I won't be discussing these two formats.
DXT5
This format can be a cause of confusion in T3D. T3D currently treats any DXT5 texture that is marked for use as a normal map as a special DXT5nm format. If you don't supply this texture in the correct format you will get very strange results indeed. This format differs from the rest by taking advantage of a great trick available with the DXT5 format. The DXT5 format has 4 channels and is laid out like so: Red 5bits | Green 6bits | Blue 5bits | Alpha 8bits.
So what we do here is take advantage of the fact that the alpha and the green channel have the highest bit count. We dump the blue (or Z) channel and we can re-create this value in our pixel shader, great now we are left with only two channels we need to worry about. So we take advantage of that alpha channel and move the red channel into it and leave the green channel as is (remember she has 6bits). Now in the pixel shader we have to do a swizzle trick that reads in the alpha and green channel.
//swizzle the alpha and green float4 normal = float4( normalMap.ag * 2.0 - 1.0, 0.0, 0.0 ); //re-create the z normal.z = sqrt( 1.0 - dot( normal.xy,normal.xy ) );
Now to get the best rest results when using this nifty DXT5 trick, we really need to fill the vacant Red and Blue channel with the same solid color (DXT5_xGxR above). One thing that the NVidia DDS photoshop plugin does by default, is when you select DXT5NM it leaves the red/blue channel as is (DXT5NM above). This produces results that are not as good because of the way DXT5 is compressed. The other format is DXT5_oGxR, currently T3D has a class called DXT5nmSwizzle that when used it writes 0's to the red channel and 1's to the blue channel. Again this doesn't allow the best quality available but is better than the default photoshop way (actually this is the format if you happened to use my PR github.com/GarageGames/Torque3D/pull/715 for enabling parallax support with DXT5).
3Dc/BC5
Now finally we have the awesome 3Dc (aka Block Compression 5 (BC5) in Direct3D10 or later). The results are far better than any of the other compressed formats. I have added support for this in my own local repo just for testing it out, there are a few problems associated with adding this to T3D but that is a discussion for another day. This format also uses an almost identical shader trick as DXT5 does.
You can find the original high resolution screenshots here narivtech.com/downloads/results.zip and the normal maps here narivtech.com/downloads/normal_maps.zip
#2
07/06/2014 (3:27 am)
Aren't there any application which can convert from .PNG to these mambo jambo formats automagically? :P 
#3
07/06/2014 (6:08 am)
great info, thanks :) 
#4
07/06/2014 (12:53 pm)
Installing the DirectX SDK provides a converter integrated into the Windows Shell, which handles most formats (DXT1,2,3,4,5) but not very specific ones (DXT5nm, etc.) 
#5
07/06/2014 (1:09 pm)
As far as NM goes, that particular case is due to it being a semi-standard usage, but not an alternate format per-se (No FourCC Identifier) Same reason the engine has to make assumptions based on where you're using dxt5s.
#6
developer.amd.com/tools-and-sdks/archive/legacy-cpu-gpu-tools/the-compressonator...
Just so people know ATI2N is 3Dc/BC5 and ATI1N is BC4. I guess now in the days of D3D10+ tools such as this are not as useful because the DXT5 trick is pretty irrelevant now (well desktop land anyway) as BC5 is a standard format in D3D10+ and OpenGL 3.0
07/06/2014 (4:53 pm)
AMD have a great tool for this called the compressonator, this is from the Ati days and for some reason they stopped developing it. It works great for getting the correct normal map formats and works equally well for any compression and mip map generation.developer.amd.com/tools-and-sdks/archive/legacy-cpu-gpu-tools/the-compressonator...
Just so people know ATI2N is 3Dc/BC5 and ATI1N is BC4. I guess now in the days of D3D10+ tools such as this are not as useful because the DXT5 trick is pretty irrelevant now (well desktop land anyway) as BC5 is a standard format in D3D10+ and OpenGL 3.0
#7
Maybe your version isn't up to date?
07/06/2014 (5:30 pm)
Unless I am misunderstanding your comment about Photoshop, saving as dxt5nm format does in fact write a copy of whats in the green channel to red and blue. Maybe your version isn't up to date?
#8
07/06/2014 (5:40 pm)
Yes it does copy what is in the green channel to the red and blue channel, this is not what you want it to do for optimal results when using DXT5 for normal maps. If you are not using the other channels for say height or anything else, the red/blue channel should be a solid color. This is why DXT5_xGxR in those screenshots produces less banding compared to the photoshop method DXT5nm (i should really have called this DXT5NM_Photo to avoid confusion)
#9
Here is what CryEngine does with normal maps
When they talk about different shaders, the only difference between the two is
To avoid the shader difference, the compressantor (above pic)has the option ATI2N alternate XY swizzle which means you can use the same shader code for both formats.
07/06/2014 (6:27 pm)
I should point out the reason you get better results with a solid color in red/blue channel is because DXT5 compresses the alpha channel separately and when it compresses the red/green/blue channels it compares the differences between the three channels, so the same solid color for the unused channels will produce better results.Here is what CryEngine does with normal maps
Quote:
Usually the Resource Compiler compresses all normal maps in the 3Dc format (1 byte per texel). When no 3Dc hardware support is available, some graphics drivers can emulate it by converting back the texture to a different format (e.g. U8V8, 2 bytes per texel). On older hardware that does not have 3Dc support at all (e.g. ATI 9800), the engine will convert normal maps to the DXT5 format at loading time. However, using the DXT5 format requires different shader combinations and the quality is a bit worse compared to 3Dc.
When they talk about different shaders, the only difference between the two is
//swizzle the alpha and green for DXT5 - Alpha/Green float4 normal = float4( normalMap.ag * 2.0 - 1.0, 0.0, 0.0 ); //don't need to swap around channels with 3Dc - Red/Green float4 normal = float4( normalMap.rg * 2.0 - 1.0, 0.0, 0.0 ); //re-create the Z is the same for both
To avoid the shader difference, the compressantor (above pic)has the option ATI2N alternate XY swizzle which means you can use the same shader code for both formats.
#10
Entirely depending on the application of course; DXT1 use much less memory space then other formats. 50% comparing to DXT5. That means you can double the normal map size for the same memory use. So if the distortion is acceptable, then you can have more detail in the normals without having a greater performance hit. Personally, I find this very useful for terrain textures like grass and such; where the pixel size matters much more then the distortion.
But I understand, it's a bit off topic; sorry :)
07/06/2014 (9:59 pm)
Quote:I'm sure everyone has noticed what a horrible format DXT1/DXT3 is for normal maps
Entirely depending on the application of course; DXT1 use much less memory space then other formats. 50% comparing to DXT5. That means you can double the normal map size for the same memory use. So if the distortion is acceptable, then you can have more detail in the normals without having a greater performance hit. Personally, I find this very useful for terrain textures like grass and such; where the pixel size matters much more then the distortion.
But I understand, it's a bit off topic; sorry :)
#11
All comments and different methods people use are all helpful to anyone reading.
07/06/2014 (10:09 pm)
Nah not off topic at all, this is actually a good feature of T3D because it's flexible about what formats one can use. Anything from nasty jpg through to dxt5nm. Some big game engines don't give you much choice in this matter and you are forced to use whatever format they deem the best.All comments and different methods people use are all helpful to anyone reading.
#12
07/06/2014 (10:46 pm)
Nice post. Also DXT1 doesn't have an alpha channel though (its the one that can only have a 1bit channel right?), so stops you using parallax, fine if youre happy with just the added detail from the normal for terrain materials.
#13
1024 x 1024
RGBA: 5.32mb
DXT1: 0.62mb
DXT5: 1.33mb
2048 x 2048
RGBA: 21.28mb
DXT1: 2.66mb
DXT5: 5.32mb
RGBA would represent a format such as png/tga/jpg etc. I don't bother listing RGB size because it's very, very likely it will be padded by your driver to sit nicely in 32bit aligned memory.
07/06/2014 (10:52 pm)
Just to expand on what nils mentions, here are texture sizes of various formats, all using mip maps(hope my calculations are ok):1024 x 1024
RGBA: 5.32mb
DXT1: 0.62mb
DXT5: 1.33mb
2048 x 2048
RGBA: 21.28mb
DXT1: 2.66mb
DXT5: 5.32mb
RGBA would represent a format such as png/tga/jpg etc. I don't bother listing RGB size because it's very, very likely it will be padded by your driver to sit nicely in 32bit aligned memory.
#14
Yep that's right, currently there is no option in T3D to use compressed normal maps with parallax mapping, that is why i put in that PR to enable dxt5 to use the red channel for height.
07/06/2014 (11:05 pm)
@Andy:Yep that's right, currently there is no option in T3D to use compressed normal maps with parallax mapping, that is why i put in that PR to enable dxt5 to use the red channel for height.
#15
You gave CryEngine as an example. With this engine you'll be able to manage the compression during optimisation before production to different platforms. It's automated with a batch process. This will speed up the process, but on the other hand are you forced to use the CryTiff format during development (a real bitch if you're porting from another engine). T3D is indeed much more flexible, but it takes more time optimising your assets.
Indeed no parallax. I meant using it in cases where the detail is more important... like plain grass for example; you'd want to focus on the detail rather then effect of height. For smooth river stones (to name another example) where you perhaps want to use parallax, DXT1 would give you to much distortion for the smooth surfaces anyway; leaving you with bad-looking compression artefacts.
07/07/2014 (1:28 am)
Quote:@Timmy: Some big game engines don't give you much choice in this matter and you are forced to use whatever format they deem the best.
You gave CryEngine as an example. With this engine you'll be able to manage the compression during optimisation before production to different platforms. It's automated with a batch process. This will speed up the process, but on the other hand are you forced to use the CryTiff format during development (a real bitch if you're porting from another engine). T3D is indeed much more flexible, but it takes more time optimising your assets.
Quote:@Andy: Also DXT1 doesn't have an alpha channel though (its the one that can only have a 1bit channel right?), so stops you using parallax, fine if youre happy with just the added detail from the normal for terrain materials.
Indeed no parallax. I meant using it in cases where the detail is more important... like plain grass for example; you'd want to focus on the detail rather then effect of height. For smooth river stones (to name another example) where you perhaps want to use parallax, DXT1 would give you to much distortion for the smooth surfaces anyway; leaving you with bad-looking compression artefacts.
#16
In a project i have just started, i'm in the middle of adding a custom texture format. When i say custom it is saved with a *.tex" extension but is just a regular dds file. This way any texture that is loaded gets a message box asking about it, so it can convert to the appropriate format using Nvidia Texture Tools(MIT license and supports 3Dc and other formats if needed later on). You can globally set a flag in script to force it to skip dds files, it will just assume you know what you are doing and just copy it the .tex extension, if you get it wrong than undefined results happen. It kind of acts like the mesh importer does. The importer box makes it clear uncompressed RGBA formats take up often double the memory or more, so unless it's absolutely critical, don't use it.
In the end this method forces the following formats(all dds surface):
RGBA: Uncompressed
DXT1-5: Compressed
Normal maps:
RGBA: Uncompressed,parallax yes
DXT1: Compressed,parallax no
DXT3: Compressed,parallax yes
DXT5_xGxR: Compressed,parallax no
DXT5_hGxR: Compressed, parallax yes (h for height)
3Dc_swizzled: Compressed, parallax no (swizzled to use same shader as DXT5_*)##
Hopefully it works ok, it gives the artist at least a bit of freedom while trying to choose the best performance.
##: This is a special format, because T3D currently uses Direct3D9 it can't be assumed 3Dc support is there. As a consequence, everytime a 3Dc texture is loaded it will first check if 3Dc is supported and if not will convert to DXT5_xGxR in real-time. This could add to loading times if no support for 3Dc. It would be very surprising to come across any hardware these days that doesn't support 3Dc though.
When complete i would be happy to give the code back to T3D but i don't know if this is something T3D really needs, it really can change from project to project.
07/07/2014 (3:47 am)
Quote:T3D is indeed much more flexible, but it takes more time optimising your assets
In a project i have just started, i'm in the middle of adding a custom texture format. When i say custom it is saved with a *.tex" extension but is just a regular dds file. This way any texture that is loaded gets a message box asking about it, so it can convert to the appropriate format using Nvidia Texture Tools(MIT license and supports 3Dc and other formats if needed later on). You can globally set a flag in script to force it to skip dds files, it will just assume you know what you are doing and just copy it the .tex extension, if you get it wrong than undefined results happen. It kind of acts like the mesh importer does. The importer box makes it clear uncompressed RGBA formats take up often double the memory or more, so unless it's absolutely critical, don't use it.
In the end this method forces the following formats(all dds surface):
RGBA: Uncompressed
DXT1-5: Compressed
Normal maps:
RGBA: Uncompressed,parallax yes
DXT1: Compressed,parallax no
DXT3: Compressed,parallax yes
DXT5_xGxR: Compressed,parallax no
DXT5_hGxR: Compressed, parallax yes (h for height)
3Dc_swizzled: Compressed, parallax no (swizzled to use same shader as DXT5_*)##
Hopefully it works ok, it gives the artist at least a bit of freedom while trying to choose the best performance.
##: This is a special format, because T3D currently uses Direct3D9 it can't be assumed 3Dc support is there. As a consequence, everytime a 3Dc texture is loaded it will first check if 3Dc is supported and if not will convert to DXT5_xGxR in real-time. This could add to loading times if no support for 3Dc. It would be very surprising to come across any hardware these days that doesn't support 3Dc though.
When complete i would be happy to give the code back to T3D but i don't know if this is something T3D really needs, it really can change from project to project.
#17
Sounds good @Timmy. They should be praised to work with a developer like you :)
Speaking as an artists I would want to make and use custom profiles for NVIDIA's DDS tool. Especially if you're working with all the kinds of different maps like diffuse, detail, specular etc. I learned over time that every application needs specific settings like mip map filtering, bias output, amount of mip maps etc.
Would be great for those who start from scratch. But I'm afraid there aren't that many nowadays :(
07/07/2014 (6:15 pm)
Quote:This way any texture that is loaded gets a message box asking about it, so it can convert to the appropriate format using Nvidia Texture Tools(MIT license and supports 3Dc and other formats if needed later on).
Sounds good @Timmy. They should be praised to work with a developer like you :)
Speaking as an artists I would want to make and use custom profiles for NVIDIA's DDS tool. Especially if you're working with all the kinds of different maps like diffuse, detail, specular etc. I learned over time that every application needs specific settings like mip map filtering, bias output, amount of mip maps etc.
Quote:When complete i would be happy to give the code back to T3D but i don't know if this is something T3D really needs, it really can change from project to project.
Would be great for those who start from scratch. But I'm afraid there aren't that many nowadays :(
#18
Yep that is definitely a huge problem with this technique, it really is something that needs to be done at the start of a project not half way through or worse near completion. When it's complete, i'll put all the code on-line for it. It may help others who are just starting a project and they can modify it to suit their own project needs.
07/07/2014 (6:41 pm)
Quote:Would be great for those who start from scratch. But I'm afraid there aren't that many nowadays :(
Yep that is definitely a huge problem with this technique, it really is something that needs to be done at the start of a project not half way through or worse near completion. When it's complete, i'll put all the code on-line for it. It may help others who are just starting a project and they can modify it to suit their own project needs.
#19
07/07/2014 (7:14 pm)
Isn't it possible to just make dxt5 work with height in alpha channel? I am pretty satisfied with dxt5, if it would work like it should. 
Timmy01
Move the red channel to the alpha channel, now paint the red channel and blue channel the same solid color (i always use black). Now save this as a regular DXT5 (not the DXT5nm). Maybe there is a way automate this via scripting in photoshop to make it a little easier?