Game Development Community

How straight-forward is it to invert vertex normals in a shader?

by Stefan Lundmark · in Torque Game Engine Advanced · 11/25/2008 (9:51 am) · 4 replies

Been trying to invert vertex normals in a vertex shader here, but simply doing the following doesn't work:

#define IN_HLSL
#include "shdrConsts.h"

//---------------------------------------------------------------
// Constants
//---------------------------------------------------------------
struct Appdata
{
    float4 position        : POSITION;
    float4 texCoord        : TEXCOORD0;
	float3 normal          : NORMAL;

};
struct Conn
{
   float4 HPOS             : POSITION;
   float2 outTexCoord     : TEXCOORD0;
   float3 normal		  : TEXCOORD1;

};
//---------------------------------------------------------------
// Main
//---------------------------------------------------------------
Conn main( Appdata In, uniform float4x4 modelview : register(VC_WORLD_PROJ),
			     uniform float4x4 texMat          : register(VC_TEX_TRANS1)

 )
{
   Conn Out;

   //take vert and get coord by transforming by modelviewmatrix
   Out.HPOS = mul(modelview, In.position);
   [b]float3 N = normalize(In.normal);
   Out.normal = -N;[/b]
   //set base texture coord
   Out.outTexCoord = mul(texMat, In.texCoord);
   return Out;
}

Do I need to touch HPOS (POSITION) at all?

#1
11/25/2008 (10:22 am)
You shouldn't need to. That should give you the inverse normal, in object space. Make sure you want it in object space.
#2
11/25/2008 (12:17 pm)
I just want the shape to be inside-out (like in art tools when you "flip normals").

The vertex shader above applied to a shape doesn't make it look any different, so I'm guessing I missed something. Perhaps I should try multiplying it with modelview..

Edit: Even setting normal to (0, 0, 0) makes no visible changes. (?)
#3
11/25/2008 (12:47 pm)
Polygons back and front face isn't determined by vertex normal. They're based on which order you specify the vertices in your triangle (clockwise vs counter clockwise) so you probably can't do that in a shader. Why not just disable front face drawing and enable drawing of the back faces?
#4
11/25/2008 (1:01 pm)
Quote:
clockwise vs counter clockwise...

This got me thinking about GFX->setCullMode() which takes both of these as an enumerated value, and that was indeed the way to go! Impressive. No need to touch the shader, as apparantly the windings will be the same even if I change the normal anyway.

Thank you!