Shadows and custom RenderDelegates

#1 Hey Ed.

Yea using your own custom render delegates and shadowing is not something we spent alot of time testing or documenting... its assumed that if your going down that path you know what your doing.

Start by looking at how regular Torque meshes shadow. If you follow the MeshRenderInst down into the RenderMeshMgr::render() you'll see the following code...

// If we have an override delegate then give it a 
      // chance to swap the material with another.
      if ( mMatOverrideDelegate )
      {
         mat = mMatOverrideDelegate( mat );
         if ( !mat )
         {
            j++;
            continue;
         }
      }

The ShadowMapPass class allocates some additional render bins and set the mMatOverrideDelegate to call ShadowMapPass::_overrideDelegate().

If you look in ShadowMapPass::_overrideDelegate() you can see that it attaches a "hook" on to the BaseMatInstance and uses it to return a special shadow material for rendering the shadow for that mesh.

The complexity is that we have at a minimum 4 different shadow shaders depending on the type of light/shadow type we're rendering. For instance the paraboloid shadow maps used for point lights need special encoding which the PSSM shadow doesn't need.

You can see how we're creating those materials in ShadowMaterialHook::init(). A key thing to notice there is this...

Material *shadowMat = (Material*)inMat->getMaterial();
   if ( dynamic_cast<CustomMaterial*>( shadowMat ) )
   {
      // This is a custom material... who knows what it really does, but
      // if it wasn't already filtered out of the shadow render then just
      // give it some default depth out material.
      shadowMat = MATMGR->getMaterialDefinitionByName( "AL_DefaultShadowMaterial" );
   }

This is my hack for CustomMaterials where i use a "stock" set of shadow materials/shaders and hope your geometry is fairly normal and you don't need alpha test shadows.

So IMO the best approch is to test for state->isShadowPass() in your prepRenderImage() and submit MeshRenderInst with your VB, PB, and a MatInstance created from the dummy AL_DefaultShadowMaterial.

If you really need to control the shader and not involve materials at all you'll have to hack the engine core to get at the info you need as the system was designed around the material system.

#2 Tom,
Thanks for the very good overview. Unfortunately, as you have probably guessed by your final sentence, we don't use materials. Our code requires a very custom set of shaders, and as such, I can simply set a "material". My shader wouldn't begin to know what is where.

I guess I'll be digging deep into the engine core to figure out what the special shadow material is and what it does. Then I can hopefully code that into a version of our shader that I can use during the shadow pass.

Thanks again for your continued help on this stuff.

= Ed =

#3 Edward, the shadow map outputs normalized depth to the red channel. Depending on the light and shadow type, the transform is different.

I am guessing that your shaders need specific transform functionality. I think the best thing to do, in this case, is to just implement that custom transform code in the shader gen code. I only recommend this because there are 3 different transform types for shadow maps. There is just standard perspective-depth via matrix * point for spot lights, and point-lights with cube maps. Vector lights that use the cascaded shadow map output slices of linear depth to an atlased shadow map (it's one big shadow map split into 3 or 4 sections). Point lights with paraboloid maps use a custom paraboloid transform in the vertex shader.

That code exists in Engine/source/lighting/shadowMap/ If you only use a limited set of lights with your game, it may be best to just pull the code out of the corresponding file. The HLSL feature for the paraboloid is in 'ParaboloidVertTransformHLSL::processVert' in Engine/source/shaderGen/HLSL/paraboloidHLSL.cpp

It is totally possible to use this code in your custom shaders, but if it is possible for you to implement custom transform code in a shader feature, and to just use materials for shadow maps, I think it would be an easier implementation. The ParaboloidVertTransform feature is a good base for that path, if you chose to do that. It is a feature which is only used during shadow map generation, and modifies the position output.

#4 Very thorough and informative response Tom.

#5 Pat,

Thanks for the informative response. Right now our code is very, very specific to the PC. This code is extremely performance sensitive, and while I haven't yet explored your shader feature capability to see if it is even possible to implement that way in a performant manner, right now I don't have the time to learn about it. Hopefully I can go back and revisit that in the same way that I have revisited using Torque to load our assets.

At this time however, I think it my only option to implement this quickly is to write my own shader for the render pass. Do I need to support all 3 different transform types, of can I simply implement one, if I know that this would suffice for my purposes? It just doesn't seem that hard to encode depth into the red channel, but shouldn't I be able to find your shadow shader code to use as an example?

Thanks,

= Ed =

#6 I think my IQ increases by a few points every time I read one of Ed's forum threads. Great information from everyone involved in the conversation! I hope what you're working on is something that will be publicly viewable someday Ed. :)

#7 I'll second what Joe suggested here!

#8 Well Joe, I certainly hope that our project will be publicly viewable someday, and I and my co-workers are diligent, maybe not too far in the future either ;)

= Ed =

#9

Quote:Do I need to support all 3 different transform types, of can I simply implement one, if I know that this would suffice for my purposes?

Assuming your just using the Sun PSSM shadow... yea... you could just implement that one and get sun light shadows, spot lights, and cubemap shadows as they share the same basic encoding.

The MFT_EyeSpaceDepthOut feature is the one that you care about. If you run one of the demo apps you can then do a search thru the procedural shaders in the shaders/procedural folder for the string "Eye Space Depth (Out)" you'll find the associated code generated for encoding the shadow depth. It will look something like this...

ConnectData main( VertData IN,
                  uniform float4x4 modelview       : register(C0),
                  uniform float4x4 objTrans        : register(C4),
                  uniform float3   eyePosWorld     : register(C8)
)
{
   ConnectData OUT;

   // Vert Position
   OUT.hpos = mul(modelview, float4(IN.position.xyz,1));
   
   // Eye Space Depth (Out)
   OUT.wsEyeVec = mul(objTrans, float4(IN.position.xyz,1)) - float4(eyePosWorld, 0.0);
   
   return OUT;
}

... and the pixel shader...

Fragout main( ConnectData IN,
              uniform float3    vEye            : register(C0),
              uniform float4    oneOverFarplane : register(C1)
)
{
   Fragout OUT;

   // Vert Position
   
   // Eye Space Depth (Out)
#ifndef CUBE_SHADOW_MAP
   float eyeSpaceDepth = dot(vEye, (IN.wsEyeVec.xyz / IN.wsEyeVec.w));
#else
   float eyeSpaceDepth = length( IN.wsEyeVec.xyz / IN.wsEyeVec.w ) * oneOverFarplane.x;
#endif
   OUT.col = float4(eyeSpaceDepth.rrr,1);
   

   return OUT;
}

If you look in the ShadowMapPass::ShadowMapPass() constructor you'll see a commented out block where we had a RenderObjectMgr bin which would allow you to get a custom delegate callback when shadow rendering. You can re-enable that and you can assign a different delegate in prepRenderImage() when state->isShadowPass().

There isn't a public machinisim at the moment for knowing what type of shadow is being requested. Its sort of burried in the mMatOverrideDelegate i showed you before. You can get at it via SHADOWMGR->mShadowMapPass->mActiveShadowType, but you will need to make a few public accessors to get there.