Using hardware fog

#1 Updating...

My suspects turned out to be true. The way TGE transforms things to use Z-up is the cause of this.

In dglSetFrustum():

static F32 darkToOGLCoord[16] = { 1, 0,  0, 0,
                                     0, 0, -1, 0,
                                     0, 1,  0, 0,
                                     0, 0,  0, 1 };

This interferes with the fragment fog behavior. I replaced it with a identity matrix, and the fog started behaving like normal... but as expected it seriously screwed the rendering, maing the camera be wrongly rotated and resulting in bad clipping. I'll try to get things to render normally while using a normal projeciton matrix..

#2 More updates...

I managed to get the camera transform back to normal by swapping the Y and Z axis before applying it to the modelView in GuiTSControl::onRender:

...
   ...
   ...
   //---------------------------------
   // FRAGMENT FOG
   //---------------------------------
   Point3F y, z;
   newCam.cameraMatrix.getColumn(1, &y);
   newCam.cameraMatrix.getColumn(2, &z);
   newCam.cameraMatrix.setColumn(1, z);
   newCam.cameraMatrix.setColumn(2, -y);   
   //---------------------------------
   // FRAGMENT FOG
   //---------------------------------

   newCam.cameraMatrix.inverse();   
   dglMultMatrix(&newCam.cameraMatrix);
   ...
   ...
   ...

Fog works correctly and camera is ok with this. All that is left is the clipping issue. TSShapes, the terrain and interiors are being badly clipped too close to the camera:


I removed the sky render bans and set the fog to red to make the problem more visible (the fog on DTS shapes is using a default value very close to the camera, the actual scene fog is set to a far distance, so the terrain is too close t be fogged).

I need to find out where in the code this clipping is being performed...

#3 What is it your doing?

#4 Trying to use OpenGL's per-pixel fog for TSShapes instead of TGE's fugly (and slow) fog. TGE's fog plain tints the whole shape a single solid color based on the fog intensity at the shape's pivot. The hardware fog applies for to each pixel, alowing big DTS shapes to be properly fogged, like this:


But I broke TGE's clipping, and need to find a way to fix it.

#5 More updates...

(I hope I'm not annoying anyone replying to my own posts, but seems nobody has ever tackled on this particular issue, so I feel obliged to post my advances on it.)

I am now trying to find a feasible way to make the modelview TGE uses to render be rotated so the view remains normal without having to rotate the projection transform itself (which messes up fragment depth fog), without affecting the clipping plane generation for frustum culling. If I rotate the camera matrix before it gets pushed into the modelview, the camera will be rendered at the correct direction, but the frustum's clipping planes will be created as if the camera were pointing "up", causing bad culling.

I found that the sceneState conveniently caches the modelview when it's created, allowing me to do the modelview rotation right before rendering everything in renderCurrentImages(), making the sceneState's generateClippingPlanes() generate proper clipping planes (fixes DTS clipping). But I found several classes which do direct reads to the OpenGL's modelview while rendering themselves. The terrain does this for it's self culling, and thus it'll use the rotated modelview, and generate clipping planes pointing at the wrong direction.

I'm looking into ways to resolve this chicken-egg issue without having to extensive modifications. When I find a solution I'll gladly post it, maybe as a resource. The hardware fog looks much nicer on DTS shapes, and it can improve performance, since it can mean a whole less render state changes between shapes.

#6 Cool Manoel, I, for one, certainly am not annoyed by your posts and do appreciate everything you are doing and have done for the community. Keep up the good work and thanks for all your help!

#7 Your threads are interesting to read but I have nothing to add so I stay quiet =)

#8 @Manoel, as you know from my shadow resource work, it's difficult to get community involvement in ground breaking stuff like what you are doing. Keep posting your updates though because at some future point, someone else is likely to get interested and they can get up to speed much quicker simply by reading these threads.

I think your fog solution is pretty important. 25% gain in performance is nothing to scoff at. Keep going and well done.

#9 The problem with using hardware fog is you lose all support for TGE's custom fog bank stuff, which is probably one of the main reasons TGE doesn't use hardware fog in the first place. Your best bet might be to do a proper fog coordinate integration into the TS pipeline (which I know at least one project has done previously).

Either way, keep up the good work.

#10 I finally made it, but there are a few things missing before I can resourcify it.

Long ago I implemented alpha test support into our codebase, but never found a way to treat alpha test-enabled shapes as opaque ones (removing the need to depth sort them). Now that the batch stuff is working, I think I can do it. That way the alpha-test implementation will finally become complete and might go into the same resource. Now that everything is in place, the average performance increase is 15% on my test machine. I'll check how it performs on our slower systems later.

As for the fog layers, I'm not concerned. The fragment fog is only used for TSShapes, and they already had some issues with fog layers, due to the "tint the whole model" approach. It's possible to make fog layers work with fragment fog by changing the fog's start and end distance depending on the shape's position along the fog layers.
Of course, it'll not be possible to have fog that affects only the lower half of a large DTS, but it was never possible to begin with. But large DTS models will look much nicer when fully contained on a single fog layer, or when you don't use layers at all.

#11 The tint texture, btw, should only be used for certain old, broken cards (Rage Pro is the main offender IIRC). Other cards should simply do a constant color blend which is very cheap to set.

TSE supports correct texture-based fogging of DTS shapes. It also will (as of the next update) do batching.

It should be possible to get textured fogging on DTS shapes by clever use of texgen, in which case you get all the benefits with very little overhead.

Hardware fogging tends to have charming foibles so you may not want to use it in production code.

Incidentally, for bleachers and similar structures, interiors would be the far better choice - you'd get more accurate collision, lightmaps!, and efficient rendering (as well as proper fogging w/o extra work).

#12 PS - GL_FOG_COORDINATE_EXT is used when you want to specify fog coordinates explicitly for each vert.

#13 I've checked it in debug mode, and I found the tint texture is used even on the GF 6600. It always updates the bitmap, it always maps it. I tried forcing the constant color, and it simply makes all models black.

The performance impact isn't as big as I thought it was before. It only becomes noticeable on scenes with dense fogging and lots of statics. The problem is that the standard tint fog is unsuitable for my batching method, since the fog values and fog bitmap must be re-calculated for every single static, which defeats my batching entirely (which works into reducing the setMaterial calls to the minimum possible).

And I'm not entirely sure how to use GL_FOG_COORDINATE_EXT, but from what I can understand, it means I would need to set fog values for each vertex manually while rendering them? Wouldn't that be slow?

And yeah, the bleachers were a bad example: they could've been DIFs and look exactly the same. It's a racing game, so the camera will never be close enough to them. Since our artists can pump out DTS files 10x faster than DIFs, so we limit our DIFs to structures we want lightmapped and walkable stuff only.

Also, the "new" fog code goes entirely inside the TSShapeInstance::setupFog() method. So it's very straightforward to add code in there for using the "old" fog if needed. I'll do some testing on lower end cards, and if any problem arises, I'll add some checks.

Ah, I also forgot to mention one thing nice about the batching I'm doing: it properly batches meshes with multiple materials. AFAIK, the new TSE batching doesn't do yet, but I'll happily find a way to make it work on TSE whenever the update becomes avaliable, since the usage of multiple materials per mesh tends to increase a lot in TSE, so we can use different shaders on dfferent materials (like cloth, skin and metal).