CustomMaterial crashes engine
by Markus Nuebel · in Torque Game Engine Advanced · 07/06/2006 (1:26 pm) · 6 replies
When using a CustomMaterial, the flow of rendering calls is as follows:
When MatInstance::setupPass( ) is called in step 1 for a custom material, the call is delegated to CustomMaterial::setupPass and the member mCurPass of MatInstance is NOT incremented and stays at it's default value 0.
When MatInstance::getCurPass( ) is called in step 2 the inline function
When MatInstance::setTextureStages( ) is called in step 3 the input param pass is filled with 4294967295 and is used to index the mPasses array.
This access results in an access violation and crashes the application.
I fixed the problem by changing the inline function to
With the signed value the material checks for mCurrPass > 0 are working again and preventing some out of bounds accesses.
However, I am not quite sure, if this is sufficient in all places, since MatInstance::setupPass( ) bails out for a CustomMaterial without doing some initializations that are done for normal materials.
Maybe someone form GG can have a look at this.
-- Markus
RenderInteriorMgr::render() --- MatInstance::setupPass( ) (1) --- MatInstance::getCurPass() (2) --- MatInstance::setTextureStages() (3) Crash
When MatInstance::setupPass( ) is called in step 1 for a custom material, the call is delegated to CustomMaterial::setupPass and the member mCurPass of MatInstance is NOT incremented and stays at it's default value 0.
When MatInstance::getCurPass( ) is called in step 2 the inline function
U32 getCurPass(){ return mCurPass - 1; }is executed with a value of 0 for mCurPass resulting in a return value of -1 that is casted to an U32 resulting in a signed value of 4294967295When MatInstance::setTextureStages( ) is called in step 3 the input param pass is filled with 4294967295 and is used to index the mPasses array.
This access results in an access violation and crashes the application.
I fixed the problem by changing the inline function to
S32 getCurPass(){ return mCurPass - 1; }With the signed value the material checks for mCurrPass > 0 are working again and preventing some out of bounds accesses.
However, I am not quite sure, if this is sufficient in all places, since MatInstance::setupPass( ) bails out for a CustomMaterial without doing some initializations that are done for normal materials.
Maybe someone form GG can have a look at this.
-- Markus
#2
07/10/2006 (5:55 pm)
This has exposed the larger problem that CustomMaterials will not work properly in this situation. I'm working on a fix.
#3
07/11/2006 (3:55 pm)
OK, I've got a fix for this - will go out with next CVS push soon.
#4
07/12/2006 (1:24 am)
Thanks Brian.
#5
07/12/2006 (12:36 pm)
This has exposed the larger problem that CustomMaterials will not work properly in this situation. I'm working on a fix.
#6
07/12/2006 (12:36 pm)
OK, I've got a fix for this - will go out with next CVS push soon.
Associate Kyle Carter