G-Buffer Normals and Trig Lookup Textures

(Cross posted from personal blog. Hopefully someone has a rockin' idea about this.)

I've switched to using spherical co-ordinates to encode world-space G-buffer normals. This has a lot of advantages to it, especially for 8:8:8:8 G-buffers. You can now store [Theta, Phi, DepthHi, DepthLo] in the 8:8:8:8 target. Bumping the format to 16:16:16:16 not only gives greater precision, but (depending on the depth resolution you need) you can get an extra channel for information storage. (Mmmm virtual texture bitfield?)

There are a few problems with this, though. The first is the atan2 function. On good hardware, this will not be an issue. My GeForce 8800 chews through any shader I throw at it. My Radeon x1300...not so much. Of course it's easy to make things run fast on good hardware. The challenge is making it run decently on lower end hardware. This is how I am encoding/decoding G-buffer normals:

inline float2 cartesianToSpGPU( in float3 normalizedVec )
{
   float atanYX = atan2( normalizedVec.y, normalizedVec.x );
   float2 ret = float2( atanYX / PI, normalizedVec.z );

   return POS_NEG_ENCODE( ret );
}

inline float3 spGPUToCartesian( in float2 spGPUAngles )
{
   float2 expSpGPUAngles = POS_NEG_DECODE( spGPUAngles );
   float2 scTheta;

   sincos( expSpGPUAngles.x * PI, scTheta.x, scTheta.y );
   float2 scPhi = float2( sqrt( 1.0 - expSpGPUAngles.y * expSpGPUAngles.y ), expSpGPUAngles.y );

   // Renormalization not needed
   return float3( scTheta.y * scPhi.x, scTheta.x * scPhi.x, scPhi.y );
}

Storing normal.z instead of acos( normal.z ) saves a decent chunk of encode/decode.

So I decided to try to use a lookup texture instead of calling atan2 to encode the normals. I made a 256x256 A8 texture and filled it with atan2 values. The texture can be seen, to the right. This is the code for generating the texture:

GFXTexHandle *RenderPrePassMgr::getAtan2Texture()
{
   if( mAtan2Handle.isNull() )
   {
      // Create a lookup texture to output a normalized atan2 result
      const U32 cLookupTexSz = 256;

      mAtan2Handle.set( cLookupTexSz, cLookupTexSz, GFXFormatA8, &GFXLookupTextureProfile, 1 );
      GFXLockedRect *atan2Mem = mAtan2Handle.lock();

      for( int y = 0; y < cLookupTexSz; y++ )
      {
         for( int x = 0; x < cLookupTexSz; x++ )
         {
            F32 xval = ( ( x / F32(cLookupTexSz) ) * 2.0f - 1.0f );
            F32 yval = ( ( y / F32(cLookupTexSz) ) * 2.0f - 1.0f );

            U8 &outU8 = atan2Mem->bits[y * cLookupTexSz + x];

            F32 atanRes = ( atan2( yval, xval ) + M_PI_F ) / M_2PI_F;
            U8 u8Res = mFloor( atanRes * 255.0f );
            outU8 = u8Res;
         }
      }

      mAtan2Handle.unlock();
   }

   return &mAtan2Handle;
}

There is a possible discontinuity when V is near 0.5 and U < 0.5. So if normal.y is near 0.0, and normal.x < 0.0 than you get some artifacts that won't occur if you actually call atan2. The A8 format isn't the issue (I don't think) because even if you use the actual function, you are still encoding the result to an 8-bit value. The resolution of the texture could be an issue, but doubling the resolution did not effect the error rate, in my tests.

I haven't quite figured out what to do with this yet. A G-buffer shader is going to be heavy on math, light on texture operations (Well this depends on how you are doing deferred shading. See upcoming ShaderX7!) and doing the atan2 as a texture sample can save a lot of instructions and cycles, depending on the hardware.