Torque 3D Beta 5 Bug: crash related to ProjectedShadow and DecalManager

This is happening in a completely unaltered version of Torque 3D Beta 5 Warrior Camp
I am running in Basic Lighting
Details of my graphics card as reported in the console:

Quote:
Reading Display Device information...
Primary Display Device Found:
Vendor Id: VEN_1002
Device Id: DEV_791E
Attempting to create GFX device: ATI Radeon X1200 Series (D3D9)
GFXD3D9EnumTranslate: Unassigned value in GFXD3D9TextureFormat: 11
GFXD3D9EnumTranslate: Unassigned value in GFXD3D9TextureFormat: 19
Device created, setting adapter and enumerating modes
Cur. D3DDevice ref count=1
Pix version detected: 2.000000
Vert version detected: 2.000000
Maximum number of simultaneous samplers: 16
Hardware occlusion query detected: Yes
WMIVideoInfo: DxDiag initialized
Initializing GFXCardProfiler (D3D9)
o Chipset : 'ATI Technologies Inc.'
o Card : 'Radeon X1200'
o Version : '6.14.0010.6925'
- Scanning card capabilities...
GFXCardProfiler (D3D9) - Setting capability 'autoMipMapLevel' to 1.
GFXCardProfiler (D3D9) - Setting capability 'maxTextureWidth' to 2048.
GFXCardProfiler (D3D9) - Setting capability 'maxTextureHeight' to 2048.
GFXCardProfiler (D3D9) - Setting capability 'maxTextureSize' to 2048.
GFXCardProfiler (D3D9) - Setting capability 'lerpDetailBlend' to 1.
GFXCardProfiler (D3D9) - Setting capability 'fourStageDetailBlend' to 1.

Reproducing the crash:
The crash occurs 100% of the time for me when in third-person (not entirely sure if this is a requirement since there is a projected shadow being rendered even when in first-person) and running into water. For some reason, it seems to happen much, much less frequently in Debug (took about 10 minutes of running around to get it to happen).

Callstack:
>FPS Genre Kit_DEBUG.dll!GFXD3D9Device::setTextureInternal(unsigned int textureUnit=0, const GFXTextureObject * texture=0x07fee718) Line 921 + 0x2f bytes C++
FPS Genre Kit_DEBUG.dll!GFXDevice::updateStates(bool forceSetAll=false) Line 520 C++
FPS Genre Kit_DEBUG.dll!GFXD3D9Device::drawIndexedPrimitive(GFXPrimitiveType primType=GFXTriangleList, unsigned int startVertex=0, unsigned int minIndex=0, unsigned int numVerts=22, unsigned int startIndex=0, unsigned int primitiveCount=13) Line 657 C++
FPS Genre Kit_DEBUG.dll!GFXDevice::drawPrimitive(const GFXPrimitive & prim={...}) Line 601 C++
FPS Genre Kit_DEBUG.dll!RenderMeshMgr::render(SceneState * state=0x080dc570) Line 222 C++
FPS Genre Kit_DEBUG.dll!RenderPassManager::render(SceneState * state=0x080dc570) Line 293 C++
FPS Genre Kit_DEBUG.dll!SceneState::renderCurrentImages() Line 160 C++
FPS Genre Kit_DEBUG.dll!SceneGraph::_traverseSceneTree(SceneState * pState=0x080dc570) Line 234 C++
FPS Genre Kit_DEBUG.dll!SceneGraph::renderScene(SceneState * sceneState=0x080dc570, unsigned int objectMask=4294967295) Line 196 C++
FPS Genre Kit_DEBUG.dll!PlaneReflector::updateReflection(const ReflectParams & params={...}) Line 509 C++
FPS Genre Kit_DEBUG.dll!ReflectionManager::update(float timeSlice=1.0000000, const Point2I & resolution={...}, const CameraQuery & query={...}) Line 113 C++
FPS Genre Kit_DEBUG.dll!GuiTSCtrl::onRender(Point2I offset={...}, const RectI & updateRect={...}) Line 141 C++
FPS Genre Kit_DEBUG.dll!GameTSCtrl::onRender(Point2I offset={...}, const RectI & updateRect={...}) Line 140 C++
FPS Genre Kit_DEBUG.dll!GuiCanvas::renderFrame(bool preRenderOnly=false, bool bufferSwap=true) Line 1498 C++
FPS Genre Kit_DEBUG.dll!GuiCanvas::handlePaintEvent(unsigned int did=0) Line 206 C++
FPS Genre Kit_DEBUG.dll!fastdelegate::FastDelegate1<unsigned int,void>::operator()(unsigned int p1=0) Line 993 + 0x1a bytes C++
FPS Genre Kit_DEBUG.dll!Signal<void __cdecl(unsigned int)>::trigger(unsigned int a=0) Line 323 + 0x17 bytes C++
FPS Genre Kit_DEBUG.dll!Journal::Call<Signal<void __cdecl(unsigned int)>,unsigned int>(Signal<void __cdecl(unsigned int)> * obj=0x02094e58, void (unsigned int)* method=0x10560ecd, unsigned int a=0) Line 541 + 0xa8 bytes C++
FPS Genre Kit_DEBUG.dll!JournaledSignal<void __cdecl(unsigned int)>::trigger(unsigned int a=0) Line 52 + 0x12 bytes C++
FPS Genre Kit_DEBUG.dll!GuiCanvas::paint() Line 1329 C++
FPS Genre Kit_DEBUG.dll!fastdelegate::FastDelegate0<void>::operator()() Line 908 + 0x16 bytes C++
FPS Genre Kit_DEBUG.dll!Signal<void __cdecl(void)>::trigger() Line 312 + 0x13 bytes C++
FPS Genre Kit_DEBUG.dll!Process::processEvents() Line 62 C++
FPS Genre Kit_DEBUG.dll!StandardMainLoop::doMainLoop() Line 517 + 0x5 bytes C++
FPS Genre Kit_DEBUG.dll!torque_enginetick() Line 78 + 0x5 bytes C++
FPS Genre Kit_DEBUG.dll!TorqueMain(int argc=1, const char * * argv=0x00a5eed0) Line 369 + 0x5 bytes C++
FPS Genre Kit_DEBUG.dll!torque_winmain(HINSTANCE__ * hInstance=0x00400000, HINSTANCE__ * __formal=0x00000000, char * lpszCmdLine=0x00151f88, HINSTANCE__ * __formal=0x00000000) Line 423 + 0x17 bytes C++
FPS Genre Kit_DEBUG.exe!WinMain(HINSTANCE__ * hInstance=0x00400000, HINSTANCE__ * hPrevInstance=0x00000000, char * lpszCmdLine=0x00151f88, int nCommandShow=1) Line 47 + 0x16 bytes C++
FPS Genre Kit_DEBUG.exe!__tmainCRTStartup() Line 263 + 0x2c bytes C
FPS Genre Kit_DEBUG.exe!WinMainCRTStartup() Line 182 C
kernel32.dll!7c817077()


Console spew just before the crash:

Quote:
*** Initial Control Object
Failed to create resource: [core/fonts/Arial 12 (ansi).uft]
Activating DirectInput...
Mapping string: setNumericalHealthHUD to index: 16
TerrainCellMaterial::init - Failed to create pass!
GFXD3D9Device::beginScene - Device needs to be reset, resetting device...
GFXPCD3D9Device::reset - depthstencil 182180 has 5 ref's
--- Resetting D3D Device ---
GFXD3D9Device::beginScene - Device needs to be reset, resetting device...
GFXPCD3D9Device::reset - depthstencil 182180 has 5 ref's
--- Resetting D3D Device ---
GFXD3D9Device::beginScene - Device needs to be reset, resetting device...
GFXPCD3D9Device::reset - depthstencil 182180 has 5 ref's
--- Resetting D3D Device ---
GFXD3D9Device::beginScene - Device needs to be reset, resetting device...
GFXPCD3D9Device::reset - depthstencil 182180 has 5 ref's
--- Resetting D3D Device ---
GFXD3D9Device::beginScene - Device needs to be reset, resetting device...
GFXPCD3D9Device::reset - depthstencil 182180 has 5 ref's
--- Resetting D3D Device ---
GFXD3D9Device::beginScene - Device needs to be reset, resetting device...
GFXPCD3D9Device::reset - depthstencil 182180 has 5 ref's
--- Resetting D3D Device ---
GFXD3D9Device::beginScene - Device needs to be reset, resetting device...
GFXPCD3D9Device::reset - depthstencil 182180 has 5 ref's
--- Resetting D3D Device ---
GFXD3D9Device::beginScene - Device needs to be reset, resetting device...
GFXPCD3D9Device::reset - depthstencil 182180 has 5 ref's
--- Resetting D3D Device ---
GFXD3D9Device::beginScene - Device needs to be reset, resetting device...
GFXPCD3D9Device::reset - depthstencil 182180 has 5 ref's
--- Resetting D3D Device ---
GFXD3D9Device::beginScene - Device needs to be reset, resetting device...
GFXPCD3D9Device::reset - depthstencil 182180 has 5 ref's
--- Resetting D3D Device ---
GFXD3D9Device::beginScene - Device needs to be reset, resetting device...
GFXPCD3D9Device::reset - depthstencil 182180 has 5 ref's
--- Resetting D3D Device ---
GFXD3D9Device::beginScene - Device needs to be reset, resetting device...
GFXPCD3D9Device::reset - depthstencil 182180 has 5 ref's
--- Resetting D3D Device ---
TerrainCellMaterial::init - Failed to create pass!
TerrainCellMaterial::init - Failed to create pass!
TerrainCellMaterial::init - Failed to create pass!
GFXD3D9Device::beginScene - Device needs to be reset, resetting device...
GFXPCD3D9Device::reset - depthstencil 182180 has 5 ref's
--- Resetting D3D Device ---
GFXD3D9Device::beginScene - Device needs to be reset, resetting device...
GFXPCD3D9Device::reset - depthstencil 1b38a0 has 4 ref's

My analysis and fix:
For some reason it appears as though the game is losing its Direct3D9 device (this is a separate issue that started in Beta 5 that I haven't yet been able to figure out) which in and of itself shouldn't cause a crash.

The problem comes with the relationship between ProjectedShadows and the DecalManager. When a ProjectedShadows renders itself to a texture, it then passes a pointer to that texture to its associated DecalInstance.

When the Direct3D9 device is lost and then subsequently recovered, all texture resources are reset. Unfortunately, the DecalInstance keeps its pointer to a now bad texture and on the next render after the device recovery tries to set that texture.

The result is that we end up with a crash in GFXD3D9Device::setTextureInternal on the call to tex->getTex().

To fix this issue, I made the following changes to the DecalManager so that it loses any references to potentially bad textures when the Direct3D9 device is lost:

In decalManager.cpp:

DecalManager::DecalManager()
{
   mObjBox.minExtents.set(-1e7, -1e7, -1e7);
   mObjBox.maxExtents.set( 1e7,  1e7,  1e7);
   mWorldBox.minExtents.set(-1e7, -1e7, -1e7);
   mWorldBox.maxExtents.set( 1e7,  1e7,  1e7);

   mDataFileName = NULL;

   mTypeMask |= EnvironmentObjectType;

   mDirty = false;

   mChunkers[0] = new FreeListChunkerUntyped( SIZE_CLASS_0 * sizeof( U8 ) );
   mChunkers[1] = new FreeListChunkerUntyped( SIZE_CLASS_1 * sizeof( U8 ) );
   mChunkers[2] = new FreeListChunkerUntyped( SIZE_CLASS_2 * sizeof( U8 ) );

   // Added the following line
   GFXTextureManager::addEventDelegate( this, &DecalManager::_onTextureEvent );
}

DecalManager::~DecalManager()
{
   // Added the following line
   GFXTextureManager::removeEventDelegate( this, &DecalManager::_onTextureEvent );

   mData = NULL;
	mDecalInstanceVec.clear();

   for( U32 i = 0; i < 3; ++ i )
      delete mChunkers[ i ];
}

void DecalManager::processTick()
{
}

// Added the following function definition
void DecalManager::_onTextureEvent( GFXTexCallbackCode code )
{
   switch(code)
   {
      case GFXZombify:
      {
         for( S32 i = 0; i < mDecalInstanceVec.size(); i++ )
         {
            if( mDecalInstanceVec[i] != NULL )
            {
               mDecalInstanceVec[i]->mCustomTex = NULL;
            }
         }
         break;
      }
   }
}

bool DecalManager::_createDataFile()
{
   AssertFatal( !mData, "DecalManager::tried to create duplicate data file?" );

   // We need to construct a default file name
   char fileName[1024];
   fileName[0] = 0;
   ...

And then in decalManager.h:

void notifyDecalModified( DecalInstance *inst );

// Added the following function declaration
void _onTextureEvent( GFXTexCallbackCode code );

void _teardownTargets();

Now, this fixes the actual crash, but doesn't prevent the Direct3D9 device from being lost. This just started happening with Beta 5 and I have yet to be able to track it down. Does anyone have any ideas as to what might be causing it to happen?

Please keep in mind that this is happening in Basic Lighting with a completely unaltered version of Torque 3D Beta 5 in Warrior Camp before posting replies.

Thanks in advance for any help!

Stephane