FIX: Dynamic lighting on CustomMaterials

We've been re-lighting existing AFX effects for TGEA and we've been pestered for a while with erratically flickering lighting whenever we have more than one dynamic light shining on a dts model using CustomMaterials.

It turns out that this is only noticeable when you are mixing dynamic lights with different lighting models (something we do quite a bit) and is only obvious when the rendering order of the lights changes (as effect lighting fades in and out, light render order changes are common).

In TGEA, a dynamic light's falloff is determined by a special 3D texture that is different for each lighting model. The essence of this bug is that when dynamic lights are lighting CustomMaterials, they are frequently not setting the correct texture for their lighting model, and using the texture from the previously rendered light. If the previous light used a different lighting model, the current light ended up rendering with incorrect falloff properties.

By itself, this error is fairly subtle, but when the rendering order of the lights changes, a different falloff texture gets used and this often produces a jarring change across several lights, hence the flickering.

Yesterday, I finally pieced together this understanding of the problem, but looking at the code, I was unsure of how to fix it. So I described the bug and sent some data concerning it to Brian Richardson at GarageGames and he came up with a nice simple block of code that so far appears to fix things. So credit for the fix goes to Brian and a big kudos from me for his help.

Here's the fix:

You need to edit the file, engine/renderInstance/renderElemMgr.cpp in two places.

First add an include for customMaterial.h near the top:

#include "renderElemMgr.h"
#include "materials/matInstance.h"
[b]#include "materials/customMaterial.h"[/b]
#include "../../example/shaders/shdrConsts.h"

Then, in the RenderElemMgr::setupLights() method, add the code-block shown below just ahead of the "need to reassign the textures..." comment. (this is the bit that Brian came up with.)

lightingmat = data.lightSecondary.sgLightingTransform;
   GFX->setVertexShaderConstF(VC_LIGHT_TRANS2, (float*)&lightingmat, 4);

   [b]CustomMaterial *custMat = dynamic_cast<CustomMaterial*>(inst->matInst->getMaterial());
   if(custMat)
   {
     custMat->setLightmaps(data);
     return;
   }[/b]

   // need to reassign the textures...
   RenderPassData *pass = inst->matInst->getPass(inst->matInst->getCurPass());
   if(!pass)
     return;

And that's it. If you're mixing dynamic lights and CustomMaterials and seeing some odd flickering, this may fix it.