Toon Shading outline help

#1

Like this?

It's done as a postfx - so it won't work in Basic Lighting, but it's all script, no code. And you can alter how thick the line is.

It's all by Matt Jolly, and slightly inconveniently it's split over 2 huge forum posts.

1. Do everything Mr.Jolly tells you here
2. And then alter it by doing everything Mr.Jolly tells you here

It works from the console so it's not hooked up to a gui or other control to enable/disable it - you'll have to do that yourself.

#2 hmmm... it looks good. Is there a way to just get it to do "silhouettes" of characters rather than every polygon outline? Like the select shader, or is that only possible through selectively doing the outline?

#3 I have absolutely no idea. Maybe read up on the silhoutte outline resources - but I think they might need substantial engine changes.

#4 Then yea, I take it that it would probably have to be only selected entities. It makes sense that way.

Thanks alot Steve for posting the resource. I'll look into it.

#5 Glen, if you don't want everything to be outlined, you probably need a new render bin into which you can select what you want to see outlined. But that unfortunately needs a source change in Torque 3D.

#6 Thanks konrad. I've modified the code that steve linked me to, of course I'm just playing around with it for now. Do you know how i could tell the shader to render all the passes after every other shader runs passes on the screen so that things like water ripples and stuff will affect it?

#7 I'm not sure I understand. Would you like to apply a custom full screen shader (post effect) as the last post effect to be done? If so, take a look at the script file renderManager.cs.

Other than that, Matt's awesome resource (that served as the main pillar of the silhouette resource) could explain a lot. You can check out the silhouette resource as an example on how to create a new renderbin and render only select objects into it.

#8 The first is probably what I'm looking for Konrad. The only thing is since I don't really know torquescript atm it might prove quite a challenge. I aim to learn but learning full torquescript right now for one thing is kinda steep. I see the file and I can understand what it does, but I'm not sure how to add a new post effect to it ^^'. I did add the effect to run automatically (by setting it after the loading process: OutlineFX.enable();)

I see these lines:

DiffuseRenderPassManager.addManager( new RenderPassStateBin() { renderOrder = 0.001; stateToken = AL_FormatToken; } );

and I figure that that's what I'm looking for. I noticed that the stateToken is created using a PPE, and making a token out of it. I also think I'm gonna need to do something like this:

new RenderFormatToken(OutlineFX_Token)
   {
	  enabled = "true";
      
      copyEffect = "OutlineFX";
      
      resolveEffect = "OutlineFX";
   };

so I can add it like this:

DiffuseRenderPassManager.addManager( new RenderPassStateBin(OutlineFX_Token_Pop)              { stateToken = OutlineFX_Token; renderOrder = 1.45; processAddOrder = 1.45; } );

However when I do this it messes up the scene. I know I gotta be at least going somewhere with this, but I'm not sure what to do from here ^^'

#9 Interesting thread.

What the use of the RenderFormatToken?

#10 I figure I need to use it to somehow "layer" the post processing effect (PPE :P) to be rendered after the water effects and all. I'm a complete noob to torquescript so I'm not sure if that's what this is achieving.

In reality I'm not even sure how I can add the OutlineFX to be rendered after everything else, but that is what I'm trying to achieve atm.

#11 Can anyone help with this?

#12 If you wish to render after all,then :

renderTime = "PFXAfterBin";
renderBin = "AL_FormatToken_Pop";

#13 Thanks, I'm sure it's running after everything now, but I think I might have looked at it from the wrong perspective, maybe it should be before water and particle effects, but after models. @.@ confusing...I've done next to nothing with PPEs.

If need be I can give a screenshot, and post the edited code.

#14 Sorry for the double posts.

I think I found the issue. I found that the shader is using the prepass buffer. I'm not sure but I'm thinking that should be the backbuffer. I also think it should be returning to the backbuffer so that the outline can be used in the water shaders and other PPEs. I'm not 100% sure but if I'm on the right track can I at least get a "you're on the right track"?

Here's the Shaders

The Detect Shader (I posted it with the original "prepassBuffer" instead of the edit I tested):

#include "postFx.hlsl"
#include "shadergen:/autogenConditioners.h"

// GPU Gems 3, pg 443-444
float GetEdgeWeight(float2 uv0, in sampler2D prepassBuffer, in float2 targetSize)
{
   float2 offsets[9] = {
      float2( 0.0,  0.0),
      float2(-1.0, -1.0),
      float2( 0.0, -1.0),
      float2( 1.0, -1.0),
      float2( 1.0,  0.0),
      float2( 1.0,  1.0),
      float2( 0.0,  1.0),
      float2(-1.0,  1.0),
      float2(-1.0,  0.0),
   };
   
   
   float2 PixelSize = 0.5 / targetSize;
   
   float Depth[9];
   float3 Normal[9];
   
   for(int i = 0; i < 9; i++)
   {
      float2 uv = uv0 + offsets[i] * PixelSize;
      float4 gbSample = prepassUncondition( prepassBuffer, uv );
      //was prespassUncondition( tex2D(prepassBuffer, uv) )
      Depth[i] = gbSample.a;
      Normal[i] = gbSample.rgb;
   }
   
   float4 Deltas1 = float4(Depth[1], Depth[2], Depth[3], Depth[4]);
   float4 Deltas2 = float4(Depth[5], Depth[6], Depth[7], Depth[8]);
   
   Deltas1 = abs(Deltas1 - Depth[0]);
   Deltas2 = abs(Depth[0] - Deltas2);
   
   float4 maxDeltas = max(Deltas1, Deltas2);
   float4 minDeltas = max(min(Deltas1, Deltas2), 0.00001);
   
   float4 depthResults = step(minDeltas * 25.0, maxDeltas);
   
   Deltas1.x = dot(Normal[1], Normal[0]);
   Deltas1.y = dot(Normal[2], Normal[0]);
   Deltas1.z = dot(Normal[3], Normal[0]);
   Deltas1.w = dot(Normal[4], Normal[0]);
   
   Deltas2.x = dot(Normal[5], Normal[0]);
   Deltas2.y = dot(Normal[6], Normal[0]);
   Deltas2.z = dot(Normal[7], Normal[0]);
   Deltas2.w = dot(Normal[8], Normal[0]);
   
   Deltas1 = abs(Deltas1 - Deltas2);
   
   float4 normalResults = step(1.0, Deltas1);
   
   normalResults = max(normalResults, depthResults);
   
   return dot(normalResults, float4(1.0, 1.0, 1.0, 1.0)) * 0.25;
}

float4 main( PFXVertToPix IN, 
             uniform sampler2D prepassBuffer :register(S0),
             uniform float2 targetSize : register(C0) ) : COLOR0
{
   return GetEdgeWeight(IN.uv0, prepassBuffer, targetSize );//rtWidthHeightInvWidthNegHeight.zw);
}

The Actual Outline Shader:

#include "postFx.hlsl"
#include "shadergen:/autogenConditioners.h"

float4 main( PFXVertToPix IN, 
             uniform sampler2D edgeBuffer :register(S0), uniform sampler2D backBuffer : register(S1) ) : COLOR0
{
   float4 e = float4( tex2D( edgeBuffer, IN.uv0 ).rrr, 1.0 );
   e.r = e.g = e.b = 1.0 - e.r;// *0.2;
   /*e.b = 1.0 - e.r * 0.6;
   e.g = e.b;
   e.r = e.g;*/
   return tex2D(backBuffer, IN.uv0) * e;
}

Or is it something I should change in the actual shader definition?:

singleton GFXStateBlockData( PFX_DefaultOutlineStateBlock )
{
   zDefined = true;
   zEnable = false;
   zWriteEnable = false;
      
   samplersDefined = true;
   samplerStates[0] = SamplerClampLinear;
};

singleton ShaderData( PFX_OutlineShader )
{   
   DXVertexShaderFile 	= "shaders/common/postFx/postFxV.hlsl";
   DXPixelShaderFile 	= "shaders/common/postFx/outlineShaderP.hlsl";
         
   samplerNames[0] = "$inputTex";
   
   pixVersion = 3.0;
};

singleton ShaderData( PFX_OutlineEdgeDetectShader )
{   
   DXVertexShaderFile 	= "shaders/common/postFx/postFxV.hlsl";
   //CHANGE THE LINE BELOW
   DXPixelShaderFile 	= "shaders/common/postFx/edgeaa/edgeDetectP.hlsl";
 
   samplerNames[0] = "$inputTex";
   
   pixVersion = 3.0;
};

singleton PostEffect( OutlineFX)
{
   renderTime = "PFXAfterDiffuse";
      
   shader = PFX_OutlineEdgeDetectShader;
   stateBlock = PFX_DefaultOutlineStateBlock;
   texture[0] = "#prepass";
   target = "$outTex";
   
   new PostEffect()
   {
      shader = PFX_OutlineShader;
      stateBlock = PFX_DefaultOutlineStateBlock;
      texture[0] = "$inTex"; 
      texture[1] = "$backBuffer";
      target = "$backBuffer";
   };
};