Game Development Community

Reposted Question re: Using new shader

by Stephen Zepp · in Torque Game Engine Advanced · 10/31/2006 (5:49 pm) · 3 replies

Mod Note: this is a repost of a question placed in the TGE forums, moved to the appropriate area:

Hey Guys,


I am having a really weird problem, I am trying to take a type of shader from RenderMonkey over to TSE. Now the shader is Silhouette and it has two passes. Now with the Converting over to TSE(which I am not sure if that is totally correct either) from RenderMonkey I have this for the Vertex and the Pixel.

Pass 0:

Pixel:
#define IN_HLSL
#include "shdrConsts.h"

float4 ps_main( float4 Diff: COLOR0 ) : COLOR
{
float4 c;
c[0] = 1.0;
c[1] = 1.0;
c[2] = 1.0;
c[3] = 1.0;

return c;
}


Vertex:
#define IN_HLSL
#include "shdrConsts.h"

float4x4 view_proj_matrix : register(VC_WORLD_PROJ);

struct VS_OUTPUT
{
float4 Pos: POSITION;
};

VS_OUTPUT vs_main( float4 Pos : POSITION )
{
VS_OUTPUT Out;
Out.Pos = mul( view_proj_matrix, Pos );
return Out;
}





And for Pass 1:

Pixel:
#define IN_HLSL
#include "shdrConsts.h"

//sampler RT;

struct PS_INPUT
{
float2 TexCoord : TEXCOORD0;
};

struct PS_OUTPUT
{
float4 Color : COLOR0;
};


PS_OUTPUT ps_main( PS_INPUT In, uniform sampler2D RT : register(S0))
{
PS_OUTPUT Out = (PS_OUTPUT) 0;

// One pixel offset
const float off = 1.0 / 512.0;

// Sample neighbor pixels
float s00 = tex2D(RT, In.TexCoord + float2(-off, -off));
float s01 = tex2D(RT, In.TexCoord + float2( 0, -off));
float s02 = tex2D(RT, In.TexCoord + float2( off, -off));

float s10 = tex2D(RT, In.TexCoord + float2(-off, 0));
float s12 = tex2D(RT, In.TexCoord + float2( off, 0));

float s20 = tex2D(RT, In.TexCoord + float2(-off, off));
float s21 = tex2D(RT, In.TexCoord + float2( 0, off));
float s22 = tex2D(RT, In.TexCoord + float2( off, off));

// Sobel filter in X direction
float sobelX = s00 + 2 * s10 + s20 - s02 - 2 * s12 - s22;
// Sobel filter in Y direction
float sobelY = s00 + 2 * s01 + s02 - s20 - 2 * s21 - s22;

// Find edge, skip sqrt() to improve performance ...
float edgeSqr = (sobelX * sobelX + sobelY * sobelY);

// ... and threshold against a squared value instead.
Out.Color = 1.0-(edgeSqr > 0.07 * 0.07);
return Out;
}



Vertex:
#define IN_HLSL
#include "shdrConsts.h"

float4x4 view_proj_matrix : register(VC_WORLD_PROJ);

struct VS_INPUT
{
float4 Pos : POSITION;
float2 TexCoord : TEXCOORD0;
};

struct VS_OUTPUT
{
float4 Pos : POSITION;
float2 TexCoord : TEXCOORD0;
};

VS_OUTPUT vs_main( VS_INPUT In )
{
VS_OUTPUT Out;

In.Pos.xy = sign(In.Pos.xy);
Out.Pos = float4(In.Pos.xy, 0.0, 1.0);
// Image-space
Out.TexCoord.x = 0.5 * (1 + In.Pos.x);
Out.TexCoord.y = 0.5 * (1 - In.Pos.y);

return Out;
}




I loaded them into Shader.cs as:
new ShaderData( Contour1 )
{
DXVertexShaderFile = "shaders/contourV1.hlsl";
DXPixelShaderFile = "shaders/contourP1.hlsl";
pixVersion = 2.0;
};

new ShaderData( Contour2 )
{
DXVertexShaderFile = "shaders/contourV2.hlsl";
DXPixelShaderFile = "shaders/contourP2.hlsl";
pixVersion = 2.0;
};




And in the materials.cs I have:

new CustomMaterial(RedPass2)
{
texture[0] = "bright_red.jpg";
texture[1] = "$backbuff";
version = "2";
shader = "Contour2";
};

new CustomMaterial(Red)
{

mapTo = "bright_red.jpg";
texture[0] = "bright_red.jpg";
version = "2";
pass[0] = "RedPass2";
shader = "Contour1";

};

Right now, not sure if its the conversion from RenderMonkey to TSE problem or how we are applying the shader to the material. Any suggestions would be very helpful!

#1
11/01/2006 (8:08 pm)
Could you describe what exactly is happening to your model, as well as provide any warnings or errors that are shaderdata or material related?

We've had a couple of people look at the above, and no one sees anything completely unusual.
#2
11/02/2006 (8:07 am)
I am getting no warning or errors. I went through the console just to double check everything is getting loaded and nothing breaking. Whats happening to the model is the material we placed the shader on which was his upper body, doesn't render. So you see a model with no upper body, just legs,hands,head, and so on. I wish there was more I could say but I can't. I am thinking it has to do something with the shader or HLSL files, because when I apply a different custom shader using the same method, it works fine. Everything renders.
#3
11/02/2006 (11:30 am)
Screen shots available by chance? I know this may break your NDA issues, but possibly you could prepare the same effect against a stock TSE resource so we could get a feel for what's going on?

Based on your additional description, I would agree that the issue is the shader itself however, given the result.

One troubleshooting technique you may find useful is to make a normal material that does something similar and look at the engine generated shader functions for comparison--it may help to determine which portion of your shader(s) isn't working as expected.

Edit--Added:

It may also make sense to comment out your second pass' color conversions from a texture, and instead have it output something wildly obvious such as bright green--that would help you to break down the issue to either a shader one, or a "second pass not being called" issue. Given what you've described, it sounds as if only your first pass is being called (turning the model completely transparent since you set all values to 1), or the second pass is being called, but not having the effect you desire. Possibly the texture being sent to the second pass isn't being properly referenced, but I'm not a shader programmer myself so I can't be sure without checking with more knowledgeable shader guys :)