T3D 1.1 Beta 1 - Occlusion Query Appears to Cause Graphics Corruption on Nvidia systems.
by Jason Parker · in Torque 3D Professional · 05/28/2010 (1:24 am) · 12 replies
Build: 1.1 Beta 1
Platform: Windows XP SP3
Issues: For the last several Nvidia driver versions (starting some time after beta 5) I've been experiencing severe graphics corruption ranging to full system lockups complete with Windows bluescreens. It took a lot of tries and finally the very latest Nvidia beta driver to finally get some debug out of the situation. It appears that the occlusion query is the root of it. What was I doing? Firing rockets. Sometimes over the water, sometimes not. Corruption / lockup would occur when firing, or during rocket flight. Of note: Trying to run T3D again after this results in an error claiming it is out of memory. A check of the video memory usage using a tool such as Riva Tuner shows video memory usage being quite high. Even impossibly high for my 512mb card.
VC++ seems to blame gfxD3D9OcclusionQuery.cpp around line 59. (Before? After? Not exactly sure. It cites a different line within ten lines every time I have managed to call a break when the graphics corruption starts.)
This is the main thread stack I was able to get:
ntdll.dll!7c90e514()
[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]
gdi32.dll!77f52138()
d3d9.dll!4fdff4b6()
d3d9.dll!4fe75c3e()
d3d9.dll!4fe72b83()
> Tairis ILEP The Arena_DEBUG.dll!GFXD3D9OcclusionQuery::getStatus(bool block=true, unsigned int * data=0x00000000) Line 61 + 0x1c bytes C++
Tairis ILEP The Arena_DEBUG.dll!ReflectorBase::calcScore(const ReflectParams & params={...}) Line 132 + 0x33 bytes C++
Tairis ILEP The Arena_DEBUG.dll!PlaneReflector::calcScore(const ReflectParams & params={...}) Line 396 + 0xc bytes C++
Tairis ILEP The Arena_DEBUG.dll!ReflectionManager::update(float timeSlice=1.0000000, const Point2I & resolution={...}, const CameraQuery & query={...}) Line 141 + 0x1e bytes C++
Tairis ILEP The Arena_DEBUG.dll!GuiTSCtrl::onRender(Point2I offset={...}, const RectI & updateRect={...}) Line 250 C++
Tairis ILEP The Arena_DEBUG.dll!GameTSCtrl::onRender(Point2I offset={...}, const RectI & updateRect={...}) Line 156 C++
Tairis ILEP The Arena_DEBUG.dll!GuiCanvas::renderFrame(bool preRenderOnly=false, bool bufferSwap=true) Line 1604 C++
Tairis ILEP The Arena_DEBUG.dll!GuiCanvas::handlePaintEvent(unsigned int did=0) Line 247 C++
Tairis ILEP The Arena_DEBUG.dll!fastdelegate::FastDelegate1<unsigned int,void>::operator()(unsigned int p1=0) Line 990 + 0x1a bytes C++
Tairis ILEP The Arena_DEBUG.dll!Signal<void __cdecl(unsigned int)>::trigger(unsigned int a=0) Line 326 + 0x17 bytes C++
Tairis ILEP The Arena_DEBUG.dll!Journal::Call<Signal<void __cdecl(unsigned int)>,unsigned int>(Signal<void __cdecl(unsigned int)> * obj=0x03d3b920, void (unsigned int)* method=0x105e005c, unsigned int a=0) Line 541 + 0xa8 bytes C++
Tairis ILEP The Arena_DEBUG.dll!JournaledSignal<void __cdecl(unsigned int)>::trigger(unsigned int a=0) Line 64 + 0x12 bytes C++
Tairis ILEP The Arena_DEBUG.dll!GuiCanvas::paint() Line 1423 C++
Tairis ILEP The Arena_DEBUG.dll!fastdelegate::FastDelegate0<void>::operator()() Line 905 + 0x16 bytes C++
Tairis ILEP The Arena_DEBUG.dll!Signal<void __cdecl(void)>::trigger() Line 315 + 0x13 bytes C++
Tairis ILEP The Arena_DEBUG.dll!Process::processEvents() Line 78 C++
Tairis ILEP The Arena_DEBUG.dll!StandardMainLoop::doMainLoop() Line 543 + 0x5 bytes C++
Tairis ILEP The Arena_DEBUG.dll!torque_enginetick() Line 103 + 0x5 bytes C++
Tairis ILEP The Arena_DEBUG.dll!TorqueMain(int argc=1, const char * * argv=0x00f4da48) Line 369 + 0x5 bytes C++
Tairis ILEP The Arena_DEBUG.dll!torque_winmain(HINSTANCE__ * hInstance=0x00400000, HINSTANCE__ * __formal=0x00000000, char * lpszCmdLine=0x00151f51, HINSTANCE__ * __formal=0x00000000) Line 423 + 0x17 bytes C++
Tairis ILEP The Arena_DEBUG.exe!WinMain(HINSTANCE__ * hInstance=0x00400000, HINSTANCE__ * hPrevInstance=0x00000000, char * lpszCmdLine=0x00151f51, int nCommandShow=1) Line 47 + 0x16 bytes C++
Tairis ILEP The Arena_DEBUG.exe!__tmainCRTStartup() Line 263 + 0x2c bytes C
Tairis ILEP The Arena_DEBUG.exe!WinMainCRTStartup() Line 182 C
kernel32.dll!7c817077()
Steps to repeat:
Use a system with an Nvidia video card.
Run a mission with a water block or water plane.
Move around and fire rockets. Over water or not over water at random.
Repeat until you experience graphics corruption and/or crash.
Suggested Workaround (not a fix): Turn off occlusion query on the water block/plane.
Platform: Windows XP SP3
Issues: For the last several Nvidia driver versions (starting some time after beta 5) I've been experiencing severe graphics corruption ranging to full system lockups complete with Windows bluescreens. It took a lot of tries and finally the very latest Nvidia beta driver to finally get some debug out of the situation. It appears that the occlusion query is the root of it. What was I doing? Firing rockets. Sometimes over the water, sometimes not. Corruption / lockup would occur when firing, or during rocket flight. Of note: Trying to run T3D again after this results in an error claiming it is out of memory. A check of the video memory usage using a tool such as Riva Tuner shows video memory usage being quite high. Even impossibly high for my 512mb card.
VC++ seems to blame gfxD3D9OcclusionQuery.cpp around line 59. (Before? After? Not exactly sure. It cites a different line within ten lines every time I have managed to call a break when the graphics corruption starts.)
This is the main thread stack I was able to get:
ntdll.dll!7c90e514()
[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]
gdi32.dll!77f52138()
d3d9.dll!4fdff4b6()
d3d9.dll!4fe75c3e()
d3d9.dll!4fe72b83()
> Tairis ILEP The Arena_DEBUG.dll!GFXD3D9OcclusionQuery::getStatus(bool block=true, unsigned int * data=0x00000000) Line 61 + 0x1c bytes C++
Tairis ILEP The Arena_DEBUG.dll!ReflectorBase::calcScore(const ReflectParams & params={...}) Line 132 + 0x33 bytes C++
Tairis ILEP The Arena_DEBUG.dll!PlaneReflector::calcScore(const ReflectParams & params={...}) Line 396 + 0xc bytes C++
Tairis ILEP The Arena_DEBUG.dll!ReflectionManager::update(float timeSlice=1.0000000, const Point2I & resolution={...}, const CameraQuery & query={...}) Line 141 + 0x1e bytes C++
Tairis ILEP The Arena_DEBUG.dll!GuiTSCtrl::onRender(Point2I offset={...}, const RectI & updateRect={...}) Line 250 C++
Tairis ILEP The Arena_DEBUG.dll!GameTSCtrl::onRender(Point2I offset={...}, const RectI & updateRect={...}) Line 156 C++
Tairis ILEP The Arena_DEBUG.dll!GuiCanvas::renderFrame(bool preRenderOnly=false, bool bufferSwap=true) Line 1604 C++
Tairis ILEP The Arena_DEBUG.dll!GuiCanvas::handlePaintEvent(unsigned int did=0) Line 247 C++
Tairis ILEP The Arena_DEBUG.dll!fastdelegate::FastDelegate1<unsigned int,void>::operator()(unsigned int p1=0) Line 990 + 0x1a bytes C++
Tairis ILEP The Arena_DEBUG.dll!Signal<void __cdecl(unsigned int)>::trigger(unsigned int a=0) Line 326 + 0x17 bytes C++
Tairis ILEP The Arena_DEBUG.dll!Journal::Call<Signal<void __cdecl(unsigned int)>,unsigned int>(Signal<void __cdecl(unsigned int)> * obj=0x03d3b920, void (unsigned int)* method=0x105e005c, unsigned int a=0) Line 541 + 0xa8 bytes C++
Tairis ILEP The Arena_DEBUG.dll!JournaledSignal<void __cdecl(unsigned int)>::trigger(unsigned int a=0) Line 64 + 0x12 bytes C++
Tairis ILEP The Arena_DEBUG.dll!GuiCanvas::paint() Line 1423 C++
Tairis ILEP The Arena_DEBUG.dll!fastdelegate::FastDelegate0<void>::operator()() Line 905 + 0x16 bytes C++
Tairis ILEP The Arena_DEBUG.dll!Signal<void __cdecl(void)>::trigger() Line 315 + 0x13 bytes C++
Tairis ILEP The Arena_DEBUG.dll!Process::processEvents() Line 78 C++
Tairis ILEP The Arena_DEBUG.dll!StandardMainLoop::doMainLoop() Line 543 + 0x5 bytes C++
Tairis ILEP The Arena_DEBUG.dll!torque_enginetick() Line 103 + 0x5 bytes C++
Tairis ILEP The Arena_DEBUG.dll!TorqueMain(int argc=1, const char * * argv=0x00f4da48) Line 369 + 0x5 bytes C++
Tairis ILEP The Arena_DEBUG.dll!torque_winmain(HINSTANCE__ * hInstance=0x00400000, HINSTANCE__ * __formal=0x00000000, char * lpszCmdLine=0x00151f51, HINSTANCE__ * __formal=0x00000000) Line 423 + 0x17 bytes C++
Tairis ILEP The Arena_DEBUG.exe!WinMain(HINSTANCE__ * hInstance=0x00400000, HINSTANCE__ * hPrevInstance=0x00000000, char * lpszCmdLine=0x00151f51, int nCommandShow=1) Line 47 + 0x16 bytes C++
Tairis ILEP The Arena_DEBUG.exe!__tmainCRTStartup() Line 263 + 0x2c bytes C
Tairis ILEP The Arena_DEBUG.exe!WinMainCRTStartup() Line 182 C
kernel32.dll!7c817077()
Steps to repeat:
Use a system with an Nvidia video card.
Run a mission with a water block or water plane.
Move around and fire rockets. Over water or not over water at random.
Repeat until you experience graphics corruption and/or crash.
Suggested Workaround (not a fix): Turn off occlusion query on the water block/plane.
About the author
Recent Threads
#3
Thanks.
05/28/2010 (1:52 am)
Can you try disabling reflection on the water and let us know how it goes?Thanks.
#4
No complete fixes but a few partial ones.
05/28/2010 (6:33 am)
Take a look see here: www.torquepowered.com/community/forums/viewthread/110795/No complete fixes but a few partial ones.
#5
My work around was simple. I made a desert.
05/28/2010 (8:01 am)
I have had the EXACT same issue with the 1.0.1 Binary as well. I am running a 9800GT with 1 gig of video ram on an intel 3 gig dual core system with 2 gigs of ram. Vista OS. My work around was simple. I made a desert.
#6
@ Aun, that's what my second post covered. Debugger went from Occlusion Query to Vertex Buffers. Appears to be particle related at my best inexperienced guess.
@ Jacob, already applied every fix I could find related to this issue, still no good.
@ Ron, for now, I'm just removing the rocket launcher. At this stage, it was counter productive anyway as we spent more time shooting at each other than actually going over the terrain marking out planned features. Willing to bet I still manage to crash as I have had *one* instance where I crashed and zero rockets were being fired or in flight.
05/28/2010 (8:36 am)
@ Caylo, such as that as well as getting Gideon's normal map filling the screen, or random flashing of polygons filling the screen.@ Aun, that's what my second post covered. Debugger went from Occlusion Query to Vertex Buffers. Appears to be particle related at my best inexperienced guess.
@ Jacob, already applied every fix I could find related to this issue, still no good.
@ Ron, for now, I'm just removing the rocket launcher. At this stage, it was counter productive anyway as we spent more time shooting at each other than actually going over the terrain marking out planned features. Willing to bet I still manage to crash as I have had *one* instance where I crashed and zero rockets were being fired or in flight.
#7
I choked on my tea laughing!
05/28/2010 (8:52 am)
Quote:
My work around was simple. I made a desert.
I choked on my tea laughing!
#8
05/28/2010 (10:55 am)
bug logged Key: TQA-155
#10
For anyone having problems with this, until the next beta hits I recommend going into the source and disabling all uses of occlusion queries.
06/09/2010 (6:22 am)
Seems occlusion queries are also used for particle rendering. So the crash is probably caused by rendering the reflections of particles. Also, the occlusion query seems to be quite heavy. Here's a profile of Burg on a GF 9800GT:Profiler Data Dump: Ordered by non-sub total time - %%NSTime %% Time Invoke # Name 25.940 25.940 3213 GFXD3D9OcclusionQuery_getStatus 7.326 24.731 10211 RenderMeshMgr_render 6.422 6.472 6797 TSSkinMesh_UpdateSkin 4.216 9.049 1377768 ProcessedShaderMaterial_SetupPass 3.781 6.113 512015 TSMesh_InnerRender 2.575 4.160 694851 MatInstance_setTransforms 2.492 9.453 459 RenderPrePassMgr_render 2.473 25.153 609671 treeTraverseVisit_prepRenderImage 2.408 2.408 6797 TSMesh_CreateVBIB 2.136 3.931 524051 GFXDevice_updateStates 2.009 2.572 688884 ProcessedShaderMaterial_SetTextureStages 1.841 2.670 3761130 GenericConstBufferLayout_set 1.794 1.794 225442 GFXD3D9StateBlock_Activate 1.686 11.116 1387866 MatInstance_SetupPass 1.668 1.668 1847325 RenderBinManager_setupSGData 1.645 2.056 693933 ProcessedShaderMaterial_SetShaderConstants 1.505 1.505 1229628 RenderPassManager_addInst 1.482 16.475 512015 TSShapeInstance_MeshObjectInstance_render 1.404 3.374 694392 MatInstance_setSceneInfo 1.390 1.390 459 SwapBuffers 1.373 1.373 1377 ParticleEmitter_copyToVB_Orient 1.298 1.298 4876 RenderPassManager_Sort 1.287 17.880 501175 TSShapeInstance_Render 1.285 1.285 6529 ContainerFindObjects_Box 1.149 1.782 535119 AdvancedLightManager_setLightInfo
For anyone having problems with this, until the next beta hits I recommend going into the source and disabling all uses of occlusion queries.
#11
Ah, now that explains a few things!
06/09/2010 (6:49 am)
Quote:
Seems occlusion queries are also used for particle rendering.
Ah, now that explains a few things!
#12
06/10/2010 (1:19 pm)
Confirmed as fixed for 1.1 Beta2. 
Torque 3D Owner Jason Parker
Call stack:
ntdll.dll!7c90e514()
[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]
gdi32.dll!77f52304()
d3d9.dll!4fe00808()
d3d9.dll!4fe1c418()
d3d9.dll!4fe74e0b()
d3d9.dll!4fe1c47d()
> Tairis ILEP The Arena_DEBUG.dll!GFXD3D9VertexBuffer::lock(unsigned int vertexStart=0, unsigned int vertexEnd=156, void * * vertexPtr=0x061b4fa8) Line 114 + 0x3a bytes C++
Tairis ILEP The Arena_DEBUG.dll!GFXVertexBufferHandleBase::lock(unsigned int vertexStart=0, unsigned int vertexEnd=156) Line 90 C++
Tairis ILEP The Arena_DEBUG.dll!GFXVertexBufferHandle<GFXVertexPCT>::lock(unsigned int vertexStart=0, unsigned int vertexEnd=0) Line 134 C++
Tairis ILEP The Arena_DEBUG.dll!ParticleEmitter::copyToVB(const Point3F & camPos={...}, const ColorF & ambientColor={...}) Line 1501 + 0x12 bytes C++
Tairis ILEP The Arena_DEBUG.dll!ParticleEmitter::prepBatchRender(SceneState * state=0x0606f460, const ColorF & ambientColor={...}) Line 728 C++
Tairis ILEP The Arena_DEBUG.dll!ParticleEmitter::prepRenderImage(SceneState * state=0x0606f460, const unsigned int stateKey=25603, const unsigned int __formal=4294967295, const unsigned int __formal=4294967295) Line 708 C++
Tairis ILEP The Arena_DEBUG.dll!SceneGraph::treeTraverseVisit(SceneObject * obj=0x05fcc510, SceneState * state=0x0606f460, const unsigned int stateKey=25603) Line 323 C++
Tairis ILEP The Arena_DEBUG.dll!SceneGraph::_buildSceneTree(SceneState * state=0x0606f460, unsigned int objectMask=4294967295, SceneObject * baseObject=0x02521318, unsigned int baseZone=0, unsigned int currDepth=1) Line 186 C++
Tairis ILEP The Arena_DEBUG.dll!SceneGraph::renderScene(SceneState * sceneState=0x0606f460, unsigned int objectMask=4294967295) Line 232 C++
Tairis ILEP The Arena_DEBUG.dll!SceneGraph::renderScene(ScenePassType passType=SPT_Diffuse, unsigned int objectMask=4294967295) Line 207 C++
Tairis ILEP The Arena_DEBUG.dll!GameRenderWorld() Line 345 C++
Tairis ILEP The Arena_DEBUG.dll!GameTSCtrl::renderWorld(const RectI & updateRect={...}) Line 53 C++
Tairis ILEP The Arena_DEBUG.dll!GuiTSCtrl::onRender(Point2I offset={...}, const RectI & updateRect={...}) Line 347 C++
Tairis ILEP The Arena_DEBUG.dll!GameTSCtrl::onRender(Point2I offset={...}, const RectI & updateRect={...}) Line 156 C++
Tairis ILEP The Arena_DEBUG.dll!GuiCanvas::renderFrame(bool preRenderOnly=false, bool bufferSwap=true) Line 1604 C++
Tairis ILEP The Arena_DEBUG.dll!GuiCanvas::handlePaintEvent(unsigned int did=0) Line 247 C++
Tairis ILEP The Arena_DEBUG.dll!fastdelegate::FastDelegate1<unsigned int,void>::operator()(unsigned int p1=0) Line 990 + 0x1a bytes C++
Tairis ILEP The Arena_DEBUG.dll!Signal<void __cdecl(unsigned int)>::trigger(unsigned int a=0) Line 326 + 0x17 bytes C++
Tairis ILEP The Arena_DEBUG.dll!Journal::Call<Signal<void __cdecl(unsigned int)>,unsigned int>(Signal<void __cdecl(unsigned int)> * obj=0x03d3b920, void (unsigned int)* method=0x105e005c, unsigned int a=0) Line 541 + 0xa8 bytes C++
Tairis ILEP The Arena_DEBUG.dll!JournaledSignal<void __cdecl(unsigned int)>::trigger(unsigned int a=0) Line 64 + 0x12 bytes C++
Tairis ILEP The Arena_DEBUG.dll!GuiCanvas::paint() Line 1423 C++
Tairis ILEP The Arena_DEBUG.dll!fastdelegate::FastDelegate0<void>::operator()() Line 905 + 0x16 bytes C++
Tairis ILEP The Arena_DEBUG.dll!Signal<void __cdecl(void)>::trigger() Line 315 + 0x13 bytes C++
Tairis ILEP The Arena_DEBUG.dll!Process::processEvents() Line 78 C++
Tairis ILEP The Arena_DEBUG.dll!StandardMainLoop::doMainLoop() Line 543 + 0x5 bytes C++
Tairis ILEP The Arena_DEBUG.dll!torque_enginetick() Line 103 + 0x5 bytes C++
Tairis ILEP The Arena_DEBUG.dll!TorqueMain(int argc=1, const char * * argv=0x00f4da48) Line 369 + 0x5 bytes C++
Tairis ILEP The Arena_DEBUG.dll!torque_winmain(HINSTANCE__ * hInstance=0x00400000, HINSTANCE__ * __formal=0x00000000, char * lpszCmdLine=0x00151f51, HINSTANCE__ * __formal=0x00000000) Line 423 + 0x17 bytes C++
Tairis ILEP The Arena_DEBUG.exe!WinMain(HINSTANCE__ * hInstance=0x00400000, HINSTANCE__ * hPrevInstance=0x00000000, char * lpszCmdLine=0x00151f51, int nCommandShow=1) Line 47 + 0x16 bytes C++
Tairis ILEP The Arena_DEBUG.exe!__tmainCRTStartup() Line 263 + 0x2c bytes C
Tairis ILEP The Arena_DEBUG.exe!WinMainCRTStartup() Line 182 C
kernel32.dll!7c817077()
I'm giving up for the night. Starting to think the first report I made is now useless and wondering how useful this one is. It appears I'm catching it *after* the really bad stuff is happening, but I can't be sure... I've not done much debugging of my own.