Normal Map Formats
by Timmy01 · 07/05/2014 (7:19 pm) · 26 comments
Currently we have a few different options to supply a normal map. Let's check out the quality first and than I'll explain the formats.
PNG
DXT1
DXT3
DXT5nm
DXT5_xGxR
DXT5_oGxR
3Dc
Before we get into any explanations, I'm sure everyone has noticed what a horrible format DXT1/DXT3 is for normal maps. I won't be discussing these two formats.
DXT5
This format can be a cause of confusion in T3D. T3D currently treats any DXT5 texture that is marked for use as a normal map as a special DXT5nm format. If you don't supply this texture in the correct format you will get very strange results indeed. This format differs from the rest by taking advantage of a great trick available with the DXT5 format. The DXT5 format has 4 channels and is laid out like so: Red 5bits | Green 6bits | Blue 5bits | Alpha 8bits.
So what we do here is take advantage of the fact that the alpha and the green channel have the highest bit count. We dump the blue (or Z) channel and we can re-create this value in our pixel shader, great now we are left with only two channels we need to worry about. So we take advantage of that alpha channel and move the red channel into it and leave the green channel as is (remember she has 6bits). Now in the pixel shader we have to do a swizzle trick that reads in the alpha and green channel.
Now to get the best rest results when using this nifty DXT5 trick, we really need to fill the vacant Red and Blue channel with the same solid color (DXT5_xGxR above). One thing that the NVidia DDS photoshop plugin does by default, is when you select DXT5NM it leaves the red/blue channel as is (DXT5NM above). This produces results that are not as good because of the way DXT5 is compressed. The other format is DXT5_oGxR, currently T3D has a class called DXT5nmSwizzle that when used it writes 0's to the red channel and 1's to the blue channel. Again this doesn't allow the best quality available but is better than the default photoshop way (actually this is the format if you happened to use my PR github.com/GarageGames/Torque3D/pull/715 for enabling parallax support with DXT5).
3Dc/BC5
Now finally we have the awesome 3Dc (aka Block Compression 5 (BC5) in Direct3D10 or later). The results are far better than any of the other compressed formats. I have added support for this in my own local repo just for testing it out, there are a few problems associated with adding this to T3D but that is a discussion for another day. This format also uses an almost identical shader trick as DXT5 does.
You can find the original high resolution screenshots here narivtech.com/downloads/results.zip and the normal maps here narivtech.com/downloads/normal_maps.zip
PNG
DXT1
DXT3
DXT5nm
DXT5_xGxR
DXT5_oGxR
3Dc
Before we get into any explanations, I'm sure everyone has noticed what a horrible format DXT1/DXT3 is for normal maps. I won't be discussing these two formats.
DXT5
This format can be a cause of confusion in T3D. T3D currently treats any DXT5 texture that is marked for use as a normal map as a special DXT5nm format. If you don't supply this texture in the correct format you will get very strange results indeed. This format differs from the rest by taking advantage of a great trick available with the DXT5 format. The DXT5 format has 4 channels and is laid out like so: Red 5bits | Green 6bits | Blue 5bits | Alpha 8bits.
So what we do here is take advantage of the fact that the alpha and the green channel have the highest bit count. We dump the blue (or Z) channel and we can re-create this value in our pixel shader, great now we are left with only two channels we need to worry about. So we take advantage of that alpha channel and move the red channel into it and leave the green channel as is (remember she has 6bits). Now in the pixel shader we have to do a swizzle trick that reads in the alpha and green channel.
//swizzle the alpha and green float4 normal = float4( normalMap.ag * 2.0 - 1.0, 0.0, 0.0 ); //re-create the z normal.z = sqrt( 1.0 - dot( normal.xy,normal.xy ) );
Now to get the best rest results when using this nifty DXT5 trick, we really need to fill the vacant Red and Blue channel with the same solid color (DXT5_xGxR above). One thing that the NVidia DDS photoshop plugin does by default, is when you select DXT5NM it leaves the red/blue channel as is (DXT5NM above). This produces results that are not as good because of the way DXT5 is compressed. The other format is DXT5_oGxR, currently T3D has a class called DXT5nmSwizzle that when used it writes 0's to the red channel and 1's to the blue channel. Again this doesn't allow the best quality available but is better than the default photoshop way (actually this is the format if you happened to use my PR github.com/GarageGames/Torque3D/pull/715 for enabling parallax support with DXT5).
3Dc/BC5
Now finally we have the awesome 3Dc (aka Block Compression 5 (BC5) in Direct3D10 or later). The results are far better than any of the other compressed formats. I have added support for this in my own local repo just for testing it out, there are a few problems associated with adding this to T3D but that is a discussion for another day. This format also uses an almost identical shader trick as DXT5 does.
You can find the original high resolution screenshots here narivtech.com/downloads/results.zip and the normal maps here narivtech.com/downloads/normal_maps.zip
#22
Again I assume this is purposely given Nvidias writeup and the fact they are not exact copies of one another. Are you sure the torque engine isn't simply doing this step wrong?
Per dx9, the last os that used it has been depreciated. I hope you reconsider adding 3dc to the main branch. Let the project tram have that option. They simply can dissallow their artists to use it if they are focused on the lowest common denominator.
07/08/2014 (2:06 pm)
Timmy your response has me more confused than ever. The writeup on Nvidia I linked before seemed to imply that for best results an engine should also look at r and b with more vector information than from g alone. That it was up to the engine developer to take this extra step. Hence Nvidias plugin is doing this purposely under that assumption. If you look at r and b even though they are the same bit depth and taken from the input textures 16 or 8 bit green channel, show slight differences when looking at the output channels.Again I assume this is purposely given Nvidias writeup and the fact they are not exact copies of one another. Are you sure the torque engine isn't simply doing this step wrong?
Per dx9, the last os that used it has been depreciated. I hope you reconsider adding 3dc to the main branch. Let the project tram have that option. They simply can dissallow their artists to use it if they are focused on the lowest common denominator.
#23
The fact the plugin says DXT5NM XY suggests there is no other rotational data,bias or scale that they mention in that paper, if there was those red/blue channel would look a lot more different than the green channel.
For example if you picked say a pixel 100 across and 100 down and sampled that pixel in a pixel shader, the data would be identical (or very close to that,see banding comment above) in either the red,green or blue channel using the NVidia plugin. To me that is a problem right there because if you are supposed to be getting further vector data from the other channels what is the point when they contain the exact same data as the green channel? Basically this format appears to be GGGR instead of xGxR.
Look i could be wrong on this, I will send NVidia an email and hopefully they can shed some light on the matter.
Just on another note, i tested the gimp dds plugin and dxt5nm and it saves it like so xGBR. It uses white for the red channel and leaves the Z in place. This produces quality not as good but is more flexible because it allows you to skip re-creating the Z if you choose. Still have to swizzle though. This format is still 100% compatible with T3D
07/08/2014 (6:34 pm)
Looking at the output of numerous normal maps compressed with the NVidia plugin the red/blue channels are identical in all of them and is obviously derived from the green channel. The only time i noticed any difference between the red/blue and the green channel was slightly more banding which makes sense because the green channel is 6bit compared to 5bit for the red/blue.The fact the plugin says DXT5NM XY suggests there is no other rotational data,bias or scale that they mention in that paper, if there was those red/blue channel would look a lot more different than the green channel.
For example if you picked say a pixel 100 across and 100 down and sampled that pixel in a pixel shader, the data would be identical (or very close to that,see banding comment above) in either the red,green or blue channel using the NVidia plugin. To me that is a problem right there because if you are supposed to be getting further vector data from the other channels what is the point when they contain the exact same data as the green channel? Basically this format appears to be GGGR instead of xGxR.
Look i could be wrong on this, I will send NVidia an email and hopefully they can shed some light on the matter.
Just on another note, i tested the gimp dds plugin and dxt5nm and it saves it like so xGBR. It uses white for the red channel and leaves the Z in place. This produces quality not as good but is more flexible because it allows you to skip re-creating the Z if you choose. Still have to swizzle though. This format is still 100% compatible with T3D
#24
Probably you looked at it but to be sure; did you check the "Image options" > "Image Type". I thought it made a difference in the channels, but I could be wrong. To come back to DXT1; I get in some cases less distortion when set to "Color Map" , then with "Normal Map Tangent" or "Object Space".
If it's totally in the wrong direction then nevermind ;)
07/09/2014 (5:49 am)
Quote:Looking at the output of numerous normal maps compressed with the NVidia plugin the red/blue channels are identical in all of them and is obviously derived from the green channel.
Probably you looked at it but to be sure; did you check the "Image options" > "Image Type". I thought it made a difference in the channels, but I could be wrong. To come back to DXT1; I get in some cases less distortion when set to "Color Map" , then with "Normal Map Tangent" or "Object Space".
If it's totally in the wrong direction then nevermind ;)
#26
There is currently no fall-back options nor is there any checks if hardware even supports BC5. It is for this reason i have not made an official PR for it. I'll try and fix this up so it can be added.
Hardware support for BC5 in D3D9
All D3D10+ capable cards plus:
Geforce 6+
Radeon 9500+
Intel G45+
I guess i should create a new thread for this as it will most likely get lost hidden away in this blog.
07/10/2014 (6:37 pm)
I have not done an official PR for 3Dc/BC5 support but if you want to test it out github.com/rextimmy/Torque3D/tree/bc5_support. This also contains the DXT5NM height in the red channel fix i made too. Don't stress, if you don't want to pack height data this way than simply ignore it, it will not have any effect at all on anything. BC5 does not support height.There is currently no fall-back options nor is there any checks if hardware even supports BC5. It is for this reason i have not made an official PR for it. I'll try and fix this up so it can be added.
Hardware support for BC5 in D3D9
All D3D10+ capable cards plus:
Geforce 6+
Radeon 9500+
Intel G45+
I guess i should create a new thread for this as it will most likely get lost hidden away in this blog.
Duion
We just need a format that supports full smooth alpha channel working with Torque3D, like dxt5, that is what brought up this discussion.
When you look at other engines, they use exotic formats like ATI2N and sometimes extra layers just for displacement. So maybe using an extra layer for parallax would solve this also.