Game Development Community

dev|Pro Game Development Curriculum

Stencil Shadows in TGE

by Brett Fattori · 03/29/2004 (11:04 am) · 75 comments

Download Code File

I haven't updated this code in over a year. This isn't guaranteed to work with the latest version of HEAD. Sorry.

Examples:
Buggy with shadows
Orc with shadows
More buggy with shadows
Orc close-up

From the read me file:

I'll warn you right now... this is going to be rough. There were a lot of modifications to the engine to get stencil shadows working. It's not going to be a simple drop in, compile and run. There's a ton of code that is added and/or modified in the engine. Additionally, this has only been implemented and tested on the Windows platform.

Included in this archive, you'll find:

+ stencil_032804.patch
+ shadowVolume.cc
+ shadowVolume.h
+ stencil_shadows.txt
+ optionsDlg.gui
+ readMe.txt (This file)



Going the patch route
========================
You can install the patch against the HEAD revision of the TGE. This method should be used by people familiar with using patches. You'll have to troll through the code and fix any problems from conflicts and such. Just run the patch against a fresh HEAD
revision. For example:

cd c:\torque
patch -p0 < stencil_032804.patch



Manual Route
=========================
Inside the file "stencil_shadows.txt" you'll find step-by-step modifications to the engine to add volume stencil shadows. I've attempted to give enough information to know where to put the bits of code. You only need to add/modify the code that is enclosed in the DK_911 tokens. It's going to take a while to go this route...



shadowVolume.cc and shadowVolume.h
==================================
You'll need to extract these files into your "/engine/game/" directory. Make sure you add them to the project file, under the "game" folder of the Torque.



Notes
===============
Z-fail shadows are not working. I tried -- there's some code in there that is the beginnings of Z-fail, but it's incomplete. Z-pass is working, however, quite nicely. You're going to see a framerate drop, unless you have some fast hardware. It's inevitable... I've done some optimizations to the code to try to minimize this. Plus, there are numerous extensions that were implemented to try to combat this. Interior lights do not cause shadows. I didn't get around to implementing these... I wasn't sure where to begin looking for the code that managed interior light sources.



Things to know
===================
There are some script commands for toggling stencil shadows, and setting detail levels. I have included a replacement for the "optionsDlg.gui" file that implements some of these settings.

Console Commands:
-----------------
toggleStencilShadows() - Toggles stencil shadows on/off
toggleStencilZFail() - Toggles use of Z-Fail code
toggleStencilExtTri() - Toggles use of "External Triangle" rendering method
setSilhouetteDelta(F32) - Set a value to determine detail level of object used for silhouette generation
toggleFastStencils() - [NOT IMPLEMENTED ANY MORE]

Associated Prefs:
-----------------
$pref::StencilShadow::renderZFailShadows
$pref::StencilShadow::externalTris
$pref::StencilShadow::infiniteProjection
$pref::StencilShadow::enableStencilShadows
$pref::StencilShadow::detailDelta

$pref::StencilShadow::gFastStencilRendering



Changes to the code
===========================
Please contribute any changes back to the community! This should be an ongoing project so that everyone can enjoy volume shadows in the TGE. For the moment, submit any changes to me at:

brettf AT renderengine DOT com

This is, by no means, a complete implementation (or even correct) of volume stencil shadows. There were numerous changes made during the development of this code that influenced design decisions.
Originally, the shadows we being rendered in a two-pass manner, but a faster one-pass system was chosen to keep speeds up. I expect that there are further optimizations that can be made to this code, and I hope that you'll share what you find and change!

Using the shadows
==========================
People have been asking me what the settings are for the datablock. Plus, it would be helpful to know what they do. These can be added to any object that derives from shapeBase or tsStatic. Here is a short breakdown of the items:

enableStencilShadows = true/false
Enable stencil shadows for this object.

sunShadowsOnly = true/false
If set to true, shadows will only be cast for the sun light (light zero in the light manager) The shadow generator will ignore the next two settings if this is set to true.

pointShadows = true/false
Set this to true to enable shadow casting for point light sources. The point light must be set to cast stencil shadows as well.

vectorShadows = true/false
If this is true, vector lights (other than the sun) will cause this object to cast shadows. Like point lights, the light must be set to cast stencil shadows as well.

alphaDistance true/false
Setting this to true will make the shadow vary in alpha based on distance from the light source.

shadowBaseAlpha = float
This is the amount of alpha the shadow will use. A value of 0.5 is pretty good.

updateEveryFrame = true/false
For shapes that do not animate, there isn't any real need to regen the silhouette for every single frame. Only when the object moves or the light moves. When this is true, the following occurs: if the object has changed position (not rotated, must change X,Y, or Z) or the light has changed position (both by some fixed epsilon) the shadow will update. Otherwise, the shadow's silhouette and volume don't get regenerated. For animated shapes (ones with ambient animations especially) set this to true. The shadow will regen each frame.

highDetail = true/false
Setting this to true will cause shadows to always generate at the highest distance detail for the object. In other words, based on the current LOD for the object. Setting this to false will cause the object's shadow to be generated from the distance detail that is n detail levels less than the current LOD. This means less polys in the shape will be used to generate the silhouette and shadow volume. Check the value in $pref::StencilShadow::detailDelta to see the delta between the highest current LOD and what will be used to generate the shadow. The value in the pref is between 0 and 10. The actual value used is subtracted from 10 to get how many LODs "down" it should use. ie: Setting it to 10 will use the highest (10 - 10 = 0 LODs down) Setting it to 5 will use 5 levels down (if available) (10 - 5 = 5 LODs down).

debugVolumes = true/false
If you want to look at the actual volume cast by your shape (not just the shadow) set this to true.

debugLightLines = true/false
To see which lights are effecting this object, set this to true. A line will be drawn (in the light's color, no less) from the light to the object.
Page«First 1 2 3 4 Next»
#61
06/26/2004 (12:27 am)
Everything seems to be packing and unpacking fine, but the object dosen't cast a shadows and when it goes of the screen the game crashs... it just give me a "Fatal Error"(i.e. Illigal Operation if you are using 98) and the closes the program.
#62
08/02/2004 (9:32 pm)
first error..
shadowVolume.cc
c:\torquetest\engine\game\shadowvolume.cc(966) : error C2065: 'glActiveStencilFaceEXT' : undeclared identifier
c:\torquetest\engine\game\shadowvolume.cc(1168) : error C2065: 'glDepthBoundsEXT' : undeclared identifier
c:\torquetest\engine\game\shadowvolume.cc(1643) : error C2039: 'generateStencilShadows' : is not a member of 'TSShapeInstance'
c:\torquetest\engine\ts\tsshapeinstance.h(80) : see declaration of 'TSShapeInstance'

2nd
shapeBase.cc
c:\torquetest\engine\game\shapebase.cc(2642) : error C2065: 'dglNPatchEnd' : undeclared identifier

3rd
tsStatic.cc
c:\torquetest\engine\game\tsstatic.cc(396) : error C2065: 'dglNPatchEnd' : undeclared identifier

4th
tsMesh.cc
c:\torquetest\engine\ts\tsmesh.cc(1367) : error C2065: 'dglNPatchEnd' : undeclared identifier

5th
tsShapeInstance.cc
c:\torquetest\engine\ts\tsshapeinstance.cc(1747) : error C2039: 'generateStencilShadows' : is not a member of 'TSShapeInstance'
c:\torquetest\engine\ts\tsshapeinstance.h(80) : see declaration of 'TSShapeInstance'
c:\torquetest\engine\ts\tsshapeinstance.cc(1753) : error C2065: 'mShape' : undeclared identifier
c:\torquetest\engine\ts\tsshapeinstance.cc(1753) : error C2227: left of '->details' must point to class/struct/union
c:\torquetest\engine\ts\tsshapeinstance.cc(1753) : error C2228: left of '.size' must have class/struct/union type
c:\torquetest\engine\ts\tsshapeinstance.cc(1757) : error C2227: left of '->details' must point to class/struct/union
c:\torquetest\engine\ts\tsshapeinstance.cc(1765) : error C2065: 'setStatics' : undeclared identifier
c:\torquetest\engine\ts\tsshapeinstance.cc(1768) : error C2065: 'smRenderData' : undeclared identifier
c:\torquetest\engine\ts\tsshapeinstance.cc(1768) : error C2228: left of '.currentTransform' must have class/struct/union type
c:\torquetest\engine\ts\tsshapeinstance.cc(1769) : error C2227: left of '->subShapeFirstObject' must point to class/struct/union
c:\torquetest\engine\ts\tsshapeinstance.cc(1770) : error C2227: left of '->subShapeNumObjects' must point to class/struct/union
c:\torquetest\engine\ts\tsshapeinstance.cc(1775) : error C2065: 'mMeshObjects' : undeclared identifier
c:\torquetest\engine\ts\tsshapeinstance.cc(1775) : error C2109: subscript requires array or pointer type
c:\torquetest\engine\ts\tsshapeinstance.cc(1775) : error C2228: left of '.generateStencilShadow' must have class/struct/union type
c:\torquetest\engine\ts\tsshapeinstance.cc(1780) : error C2228: left of '.currentTransform' must have class/struct/union type
c:\torquetest\engine\ts\tsshapeinstance.cc(1783) : error C2065: 'clearStatics' : undeclared identifier


so this is what I get for errors when I compile.. its confusing to me.. thought I would post and see what the experts think.. :)
#63
10/16/2004 (11:15 pm)
I'm getting the same errors as Tom Feni is. I've patched this with a clean version of 1.3 and gotten rid of most of the errors except those. I'm not sure where to put some of the code in the patch file since the line numbers for some of them do not match up with the length of the files for torque version 1.3. Any help would be appreciated.
#64
12/10/2004 (2:15 pm)
Sorrym Newb question... I want to add this to my game but add it by hand (don't know how to patch yet)
Do I have to add all the stuff from the patch file in to the engine myself?

Or is there an easy way to patch the TGE?
#65
01/10/2005 (5:05 am)
A few questions before I implement this:

1. Will this work if there is no terrain?
2. Will this allow self-shadowing?
#66
01/28/2005 (7:12 am)
Just wondering if anyone had the patch file for 1.3 version?
#67
04/04/2005 (8:43 am)
Is it compatible with 1.3 version?
#68
05/24/2005 (6:09 am)
Hi all,

Brief explanation, as GG's stupid comment system just ate my longer post (GG guys, please fix the fact that if you change the rating drop-down at all, your comment gets lost in the posting process!).

I've applied the resource, using Ben Woodhead's patch. Most of it is good, but the GL extension initialisation changed a lot since that patch was made. Short story: the extensions needed are not initialised and hence stencil shadowing won't work, even if you get no compile-time errors.

Check your console.log:

Activating the OpenGL display device...
 Missing OpenGL function 'glActiveStencilFaceEXT'
 Missing OpenGL function 'glDepthBoundsEXT'
 Missing OpenGL function 'glWeightPointerARB'

 Missing OpenGL function 'glVertexBlendARB'
You are missing some OpenGL functions.  That's bad.

I'm working on a change to initialise these extensions using the new (super elegant) GL extension system in 1.3. As soon as I get it working, I'll post an update.
#69
05/24/2005 (8:40 am)
Argh. Damn form ate my comment again. :(

The main issue with this stencil shadow implementation is that it's not supported on ATI cards. Bah.

I'm writing a separate path for ATI cards using the ATI_separate_stencil extension, which is ugly, but should work for now until I can merge the 2 paths into something a little more opengl 2.0 compliant.

If anyone has already done this, or knows a better route, please speak up.

I will have my stencil shadows, I will. :)
#70
05/25/2005 (12:25 pm)
Okay, the good news:

I've got the stencil shadows working correctly with the ATI separate stencil op extension. Works like a dream.

The bad news:

A performance hit of about 50-75% compared to using the standard shadow path for the player object with full detail shadows that update every frame.

I'm looking at alternative rendering techniques, but would love to hear from anyone else that has implemented stencil shadows in TGE as to how they're approaching the problem.

I'll post a resource for getting stencil shadows in a stock 1.3 as soon as I've tidied up and tested a bit more.
#71
05/25/2005 (12:50 pm)
@Sebastian: Sadly I don't have experience with stencil shadows in TGE. However I just really wanted to say that I'm totally thrilled that you're intending to post a resource for the work you've done. I guess this is just "Thanks" in advance!
#72
01/29/2006 (4:06 pm)
Hi, I tried patching the game, but I can
#73
08/06/2006 (6:06 pm)
I found what may be a problem putting the code in by hand.
in:stencil_shadows.txt, there is the following:
game\shapeBase.cc(2550) :: ShapeBase::renderImage()
## Add after the method ShapeBase::renderShadow() ##
=========================================================================================

// DK_911 ---------------------------------------------------------------
void ShapeBase::generateStencilShadow(F32 dist, SceneState* state)
{
	// Is this necessary
	if (mShapeInstance->getShape()->subShapeFirstTranslucentObject.empty() || mShapeInstance->getShape()->subShapeFirstTranslucentObject[0]==0)
	  return;

	dglNPatchEnd();

	// During object rendering, we're only generating the volumes
	if (!mStencilShadow)
	  mStencilShadow = new StencilShadow();

	// What lights will this object cast shadows for?
	mStencilShadow->setLightingMode(mDataBlock->mShadowSunlightOnly,mDataBlock->mShadowPointLights,mDataBlock->mShadowVectorLights);

	// Set the current scene state
	mStencilShadow->setSceneState(state);

	// Shadow debugging
	mStencilShadow->setDebug(mDataBlock->mDebugVolumes,mDataBlock->mDebugLightLines);

	// Do we care how far we are from point lights?
	mStencilShadow->setDistanceAlpha(mDataBlock->mDistanceAffectsAlpha);

	// Set the base alpha value
	mStencilShadow->setBaseAlpha(mDataBlock->mShadowBaseAlpha);

	// Update frequency
	mStencilShadow->setUpdateEveryFrame(mDataBlock->mUpdateEveryFrame);

	// Use highest detail level for shadows?
	mStencilShadow->useHighestDetail(mDataBlock->mUseHighestDetail);

	// Get position in world
	Point3F pos = mShapeInstance->getShape()->center;
	pos.convolve(mObjScale);
	mObjToWorld.mulP(pos);

	F32 maxScale = getMax(mObjScale.x,getMax(mObjScale.y,mObjScale.z));

	// Generate the volumes
	mStencilShadow->selectShapeDetail(mShapeInstance,dist * 2,maxScale);
	mStencilShadow->generateShadows(mShapeInstance,getRenderTransform(),pos,mObjScale,getWorldSphere(),getWorldBox());

	// Register volumes for rendering
	gStencilShadowRenderer->registerShadow(mStencilShadow);

}
// DK_911 ---------------------------------------------------------------

I searched for ShapeBase::renderImage() and ShapeBase::renderShadow() using the search tool in vs2005, but found no reference in shapeBase.cc so I added this to the end of the file. Hope that's where it was supposed to go.
#74
06/09/2007 (12:03 am)
Has anybody got this working in 1.5? I'm going to have a go, because it looks so beautiful...
EDIT: I'm going to need some professional help, methinks :P. Everything looked good until I got to the GL stuff. Seems like lots has changed since 1.3...
#75
10/26/2007 (3:32 pm)
We are using 1.5.2 .... my question is can the color be turned into a 'white' shadow instead of 'black'?
Page«First 1 2 3 4 Next»