Game Development Community

Graphics Corruption. Win7 + Nvidia + D3D Showstopper

by Andrew "Blorf" Hume · in Hardware Issues · 02/21/2011 (9:59 pm) · 14 replies

Reference project:
www.spazgame.com

Backstory:
I have been running TGB 1.7.4 without a hitch for 18 months, then a week ago I submitted a build to Steam and it failed, I also submitted to Rock Paper Shotgun and it failed. I was flabbergasted and personally destroyed (life savings tied up in the project) Since then, I have upgraded to TGB 1.7.5, but the problem persists.

Description:
The game runs perfectly in OpenGL on all platforms. It runs perfectly in D3D in XP/Vista, but when run on Windows7 with what seems like consistently Nvidia high end (DirectX 11 capable) cards, the game graphics are corrupted and there is an incredible slowdown when this happens. (See videos)

Interestingly, the slowdown seems related to effects that flash (like explosions) or are scaled from their base sizes. There are periods where the game runs fine when no flashing effects are playing. I made a build without any effects enabled, and it became a lot more stable in D3D on a system where we know the Windows 7 problem exists. The effects that I think are causing the issue are using the "intense" rendering. (This is just a theory. I cannot reproduce this bug, nor can my partner who uses Windows7, but has an ATI card) This bug seems related to high end Nvidia cards on windows7 (with the latest drivers)

Another possibility is that the corruption occurs when many alpha layers interact as happens with explosion particles and the starmap nebulas.

Here are links to the videos: (Sorry for the adds, it is part of the service)
http://www.justin.tv/bunkerface/b/280121731 (Shows original problem with effects running, note the graphical corruption whenever there is a bright flash, but short periods of normal play when no fighting occurs)

http://www.justin.tv/bunkerface/b/280127071 (Effects turned off, game is mostly working in FE, but major issues happen after the star generation screen)


Resolution Attempt Info:
There are no console log irregularities between working and non working builds
Upgraded to TGB 1.7.5, compiled in VS2008 with new dpi manifest
Tried running in admin mode
Tried forcing dx9 mode
Tried XP compatibility mode
Tried builds with all preloaded art
Tried builds with sound disabled (in case was an audio issue. graspinh at straws at this point)
Tried build with no effects (This resolved problems in the FE, but after starmap, graphics become scrambled)

Info from a machine that causes failure:

The windows 7 system (a dell vostro 1520 laptop)

Time of this report: 2/22/2011, 01:38:49
Operating System: Windows 7 Home Premium 32-bit (6.1, Build 7600) (7600.win7_gdr.101026-1503)
Language: Dutch (Regional Setting: Dutch)

System Manufacturer: Dell Inc.
System Model: Vostro 1520
BIOS: Ver 1.00PARTTBLd
Processor: n.v.t.
Memory: 1024MB RAM
Available OS Memory: 1018MB RAM
Page File: 1006MB used, 1036MB available
Windows Dir: C:\Windows
DirectX Version: DirectX 11
DX Setup Parameters: Not found
User DPI Setting: Using System DPI
System DPI Setting: 96 DPI (100 percent)
DWM DPI Scaling: Disabled
DxDiag Version: 6.01.7600.16385 32bit Unicode

---------------
Display Devices
---------------

Card name: NVIDIA GeForce 9300M GS
Manufacturer: NVIDIA
Chip type: GeForce 9300M GS
DAC type: Integrated RAMDAC
Device Key: Enum\PCI\VEN_10DE&DEV_06E9&SUBSYS_02BC1028&REV_A1
Display Memory: 498 MB
Dedicated Memory: 245 MB
Shared Memory: 253 MB
Current Mode: 1440 x 900 (32 bit) (59Hz)
Monitor Name: Algemeen PnP-beeldscherm
Monitor Model: unknown
Monitor Id: SEC5442
Native Mode: 1440 x 900(p) (59.930Hz)
Output Type: Internal
Driver Name: nvd3dum.dll,nvwgf2um.dll,nvwgf2um.dll

Driver File Version: 8.15.0011.8619 (English)
Driver Version: 8.15.11.8619
DDI Version: 10
Driver Model: WDDM 1.1
Driver Attributes: Final Retail
Driver Date/Size: 6/11/2009 15:39:00, 7616000 bytes
WHQL Logo'd: Yes
WHQL Date Stamp:
Device Identifier: {D7B71E3E-45A9-11CF-3352-B72201C2C535}
Vendor ID: 0x10DE
Device ID: 0x06E9
SubSys ID: 0x02BC1028
Revision ID: 0x00A1
Driver Strong Name: oem12.inf:NVIDIA_SetA_Devices.NTx86.6.1:Section026:8.15.11.8619:pci\ven_10de&dev_06e9&subsys_02bc1028
Rank Of Driver: 00E60001

The console gives no errors or warnings when these hitches occur.

Plea for help:
I am not a hardware expert, and I am at the mercy of the Torque support team here. I am willing to make my source code privately available to Garagegames as a means to identifying this problem if need be. This error happens regularly when the demo is given to people with high end machines, and it has devastated our project. Please get back to me. This issue has stopped our production until we can resolve.

I will provide any and all info needed to fix this issue. I do not have the hardware that causes the corruption, nor does my partner, but it definitely seems Windows7 + Nvidia based. Lower end Nvidia cards work fine as well. It seems just to be the new models, those that people at Steam and review sites have.

Eagerly waiting for a response,

Andrew Hume,
MinMax Games Ltd

#1
02/22/2011 (1:59 am)
Quote:The game runs perfectly in OpenGL on all platforms. It runs perfectly in D3D in XP/Vista, but when run on Windows7 with what seems like consistently Nvidia high end (DirectX 11 capable) cards, the game graphics are corrupted and there is an incredible slowdown when this happens. (See videos)

Can you force it to run only OpenGl on all OS and platforms ? I remember having some odd issues with TGB 1.74 - 1.75 on D3D also and have always force my game to run with OpenGl ever since.
#2
02/22/2011 (2:05 am)
Great looking game by the way. :)
#3
02/22/2011 (8:24 am)
Thanks for the compliment.

Unfortuanatley, OpenGL runs about 40% slower than D3D, and on most platforms D3D works great. We would be cutting out the lower end of our game market for the sake of sidestepping this issue.

This is a compatability issue that needs to be resolved for all TGB users who will be running in to this in the near future. I assume a hardware compatability specialist would be able to resolve this without too much problem. The repro on the bug is easy once you have the offending hardware and our build. (We have not changed any core functionality to TGB, so this will happen to anyone who pushes the effects system the way we have)

We are just devastated by this bug, and the fact that we didn't know it was there. We had high hopes for steam distribution which have now unfortunatley fizzled due to the bad first impression. Anyhow this thread is not about a project disaster but about getting it fixed hopefully.
#4
02/22/2011 (3:55 pm)
The info you show of the machine causing issues isn't running the current nvidia drivers. Do you have a demo I can play with? Here are my stats and would be willing to test
------------------
System Information
------------------
Time of this report: 2/22/2011, 16:54:36
       Machine name: ERIC-PC
   Operating System: Windows 7 Ultimate 64-bit (6.1, Build 7600) (7600.win7_gdr.101026-1503)
           Language: English (Regional Setting: English)
System Manufacturer: To Be Filled By O.E.M.
       System Model: To Be Filled By O.E.M.
               BIOS: Default System BIOS
          Processor: AMD Athlon(tm) II X4 645 Processor (4 CPUs), ~3.1GHz
             Memory: 8192MB RAM
Available OS Memory: 8192MB RAM
          Page File: 2474MB used, 13905MB available
        Windows Dir: C:\Windows
    DirectX Version: DirectX 11
DX Setup Parameters: Not found
   User DPI Setting: Using System DPI
 System DPI Setting: 96 DPI (100 percent)
    DWM DPI Scaling: Disabled
     DxDiag Version: 6.01.7600.16385 64bit Unicode
---------------
Display Devices
---------------
          Card name: NVIDIA GeForce GTX 460 SE
       Manufacturer: NVIDIA
          Chip type: GeForce GTX 460 SE
           DAC type: Integrated RAMDAC
         Device Key: Enum\PCI\VEN_10DE&DEV_0E23&SUBSYS_00000000&REV_A1
     Display Memory: 4065 MB
   Dedicated Memory: 993 MB
      Shared Memory: 3071 MB
       Current Mode: 1920 x 1080 (32 bit) (60Hz)
       Monitor Name: Generic PnP Monitor
      Monitor Model: 2436
         Monitor Id: AOC2436
        Native Mode: 1920 x 1080(p) (60.000Hz)
        Output Type: HDMI
        Driver Name: nvd3dumx.dll,nvwgf2umx.dll,nvwgf2umx.dll,nvd3dum,nvwgf2um,nvwgf2um
Driver File Version: 8.17.0012.6658 (English)
     Driver Version: 8.17.12.6658
        DDI Version: 10.1
       Driver Model: WDDM 1.1
  Driver Attributes: Final Retail
   Driver Date/Size: 1/7/2011 20:27:00, 12859496 bytes
        WHQL Logo'd: Yes
    WHQL Date Stamp: 
  Device Identifier: {D7B71E3E-4D63-11CF-B279-0C201FC2C535}

#5
02/22/2011 (3:56 pm)
Notes from today's attempts to track:

Works fine on Win7 x64 with ATI Card with D3D and OpenGL
Works fine Win 7 x64 with Nvidia Card (Laptop 310M) with OpenGL but not D3D

Windows Settings
Windows Aero Off + 256 Col + Desktop Comp + Scaling DPI - No Change

GPU Driver Settings
Force Antroscoptic Filtering Off - No Change
Force Anti Aliasing Off - No Change
Force Anti Aliasing On x2 - No Change
VSync Forced off - No Change + BlueScreen
VSync Force On - Better, Corruption occurs on star map, 2nd test, no change
Threaded Optimisation Off - No Change
Trilinier Optimisation - Off - No Change
Mixed GPU Acceleration - Compatibility Mode - No Change
Texture Filterin - Performance - No Change
PhysX to GPU instead of CPU - No Change
Max Pre Rendered Frames to 0 - No Change
Triple Buffering On - No Change
Ambient Occlusion - Performance - No Change
Power Management Mode - Max Performance - No Change
VSync Forced On, Screen Set to 50Hz - No Change
VSync Forced On, Screen Set to 60Hz - No Change
Negetive LOD balance Clamp - No Change

Tried setting $pref::timeManagerProcessInterval = 17; - No change

Using a build with all effects removed (to prevent as much alpha scaling and setBlending(true, SRC_ALPHA, ONE); as possible. The game got a LOT better and run until failing at the starmap. This is shown in the second video mentioned above)

Ran in PIX from MSDN, but did not get any valuable info, but here it is.
PIX Logfile created at: 23:50:20

Frame 000001 ....PRE: Frame(1)
Trigger 'Frame 1' fired
Created PIXRun file at: C:\Users\Chris\AppData\Local\Temp\PIX402D.tmp
Wrapper: Not using 888 texture
Wrapper: Not using luminance texture
Wrapper: Not using alpha texture
Wrapper: No palette textures -- no GL_EXT_paletted_texture
Wrapper: Multitexturing enabled
Closing Run File





#6
02/22/2011 (4:34 pm)
@Eric

Thanks for the offer. It looks like your machine fits the bill. Here is my email: blorf@telus.net so we can get into contact. I have an ftp server with the file.
#7
02/22/2011 (6:13 pm)
I have a hardware compatibility set up with 10+ machines including systems with Windows Starter 32 and Windows 64 based.

My main development computer has Windows 7 x64 pro with a NVIDIA 9800 GS, similar to the NVIDIA GeForce 9300M GS in the PC that shows the issue.

I am willing to test out your game on my systems to see if I can reproduce the issue and I can help fix it if it shows up. Especially since my dev. computer has VS 2008 on it already etc. (and I'm an experienced programmer).

BTW... glad to see a fellow Canadian making sweet games!

EDIT: I should also note my system dual boots to Vista so I can find out if it's a video card issue or if it's OS related if it's reproducable.
#8
02/23/2011 (9:55 am)
Chris Labombard is kindly helping me with this issue. I will update when we find out anything positive or negative.

Additional testing notes:
We have tried the build on an Nvidia video card based dual boot vista64/win7 64 system. In vista 64 the issue does not exist, but in windows 7 it does.

Also on a windows7 system with the error using an nvidia card, we installed an ATI card and the error stopped.

So the original diagnosis seems sound.
Highend Nvidia + Windows7 + D3D = Graphic corruption.
#9
02/23/2011 (7:01 pm)
Today's update.

Chris took a look at the bug on his machine and was able to reproduce it, but we have not fully isolated the issue, so ANYONE who has ideas please let us know.

Chris and I discussed and the graphical glitches, and we are thinking that it could be a symptom of the problem but not the root problem.

Since Win7 has TDR which resets the video card under heavy load, we would expect to see a graphical glitch upon reset (similar to what we are seeing) The problem may actually be that Nvidia cards are for some reason detecting a heavy load, but only windows 7 does something about it. If the game runs on Vista, the problem may still occur, but it instantly resolves itself instead of triggering a card reset so everything seems normal. This is just a guess and we are grasping at straws here.

When we disabled the effects system, the problem became a LOT less frequent. We tried disabling all calls to setBlending that used glBlendFunc(GL_SRC_ALPHA, GL_ONE) but that turned out to be a dead end.

The effects system is doing something frequently that is causing this issue, but other systems are also doing that, but much less often.

Could it have something to do with scaling textures? The effects system sets particle size quite often as they fade away. This is also a guess. The hitches do seem related to larger textures appearing and fading out though or changing their alpha. Maybe it is an issue with changing the alpha on the fly?

If you have any knowledge of the gl2d3d wrapper, it could go a long way. Please help.





#10
02/24/2011 (5:51 am)
The problem occurs rapidly when the particle effects are displaying. Some piece of rendering code that the particles rely on heavily is causing the NVIDIA graphics driver for Win 7 to stall out.

When it stalls too much Windows BSOD on the NVIDIA driver, presumably because of a failed TDR from the OS.

if anyone knows a lot about the rendering code behind the particle emitters it would be a lot of help.
#11
03/02/2011 (12:54 pm)
This bug is still present and must be fixed. Can someone suggest a current GarageGames employee that I can contact directly with this? I do not want to spam them because they are busy, but this is a very serious issue.

I am at the point now where I may have to contact NVidia directly to get some answers, however I am certain that they would take GarageGames more seriously than a lone individual.



#12
03/02/2011 (3:39 pm)
I've been watching the thread since it started, last week was crazy busy for me so I haven't been able to jump in until now.

I was thinking this may have been similar to the particle issue we had with Nvidia in T3D, but that turned out to be the particles needed to be copied to the vertex buffer every frame so that they would work correctly in reflections.

Can you send me a copy of your project to take a look at?
#13
03/02/2011 (4:34 pm)
Hi Scott,

I have sent you an email from your profile info. If the email is correct, I will send you FTP information right away.
#14
03/05/2011 (1:10 pm)
The Saga continues!

So, TGB 1.7.4 and TGB 1.7.5 will always fail in D3D mode with an Nvidia Card on Windows 7. So we are talking a large percentage of the gamer population here.

To prove this, we downloaded the Larva Mortis demo from Steam, set it to D3D rendering and we achieved the exact same failure as above. Further we also modified ArchMage to use D3D rendering (modified defaultprefs.cs) and the same problem is there.

SO, as of now if you ship your TGB game with D3D enabled, you are looking for trouble. We are fairly certain that our Steam distribution failure was caused by this bug, so beware!

I have spent the last 6 hours building a simple game using the Stock TGB 1.7.5 engine that will cause this issue. The only change to the engine is to the telnet debugger to allow Torsion to connect. Stock TGB will also work, but there will be no debugging.

I will leave this on my personal FTP site: Please download and have a look. There are on screen instructions.

Host: 173.180.58.151
User: friend
Pass: win7borked
port: 5105

The file is called win7breaker.zip (it is only 2 megs)

What win7breaker has shown us is that the graphical corruption is not solely particle based, in the sprite drawing mode, when spawning a large number of stationary sprites, the error also occurs.

Here are the game's features:

Effects mode: allows you to set the number of effects playing and how often new sets of effects are made

Sprite mode: allows you to set the number of on screen sprites

Toggle background: You can turn a background on and off with T, but we have not seen any relation to the bug either way.

You can switch between openGL and D3D rendering on the fly by pressing O and D respectively. When the game fails in D3D, you can just hit O and everything will go back to normal.

As for driver issues:

We have tried the latest Nvidia drivers, the beta Nvidia Drivers, and even going back to Nvidia drivers from 2009, the 196 series of drivers. ALL cause the same failure. I think that it is highly unlikely that NVidia will see this issue as their problem since it has existed for over a year at least.

So where do we go from here:

The win7Breaker is probably the best tool you are going to get to find this issue. There is nothing extraneous going on in the game, and creating the issue can be achieved with a few button presses. The .cs files are all included with the zip.

In talking to Scott, they agree that this is an important issue and a high priority, but also lower priority than iTGB or T3D updates, so we TGB users are currently stuck with broken D3D.

I think that the work I have done to create a very simple demo program that completely isolates the problem could give the right person a really good shot of finding this issue and in a timely manner. I hope someone somewhere will take a look.