Simple toon shader

#1 Cool Kevin! If more people are interested in posting this kind of thing, it might be a good idea to set up a specific forum for sharing shaders.

#2 Su-weet

#3 That may be a good idea Brian... Especially Later when all the noobs are screaming "How do i make a shader that does X??

-- not to say that I'm not a noob atm..:)

#4 I got the shader working on the Space Orc model and I must say it is a cool effect. The thing I don't understand however is that none of the textures for the model come through, the entire model is red like in your screenshot. If I want to change this do I need to modify the shader or is it something that I need to change in the Material datablock?

#5 Heh, oh yah.. i posted this back when i had very little idea of what was actually going on..:)

Quote:
I havent tested this but thats pretty much the way you would do it..
I should prolly revisit this and post a decent toon shader, but im swamped atm..

good luck

update 3 years later LMAO!!

#6 Hi,

I was just trying your toon shader and it kicked out an error. The part where it sais to add in the pixel shader struct INPUT
uniform sampler2D diffuseMap : register(S0);
It gives me an error: object types are not allowed in structs

Also, in the OUT.Color0 = Color_ID9 * tex2D(diffuseMap, IN.texCoord);
It gives an error: invalid subscript texCoord

Any ideas?

Thx.

#7 Missing a couple of things.. Following works with TGEA 1.7.1.

toonV.hlsl

//  pixVersion = 2.0;


#define IN_HLSL
#include "shdrConsts.h"

struct Appdata
{
	float4 IN_Vertex_Position : POSITION; //no comments
	float2 texCoord			  : TEXCOORD0;
	float4 IN_Vertex_Normal   : NORMAL;   //no comments
};


struct Conn
{
	float4 TC0_Vertex_Position : POSITION; //no comments
	float2 outTexCoord     : TEXCOORD0;
	float4 TC1_Vertex_Normal   : TEXCOORD1; //no comments
	float4 ObjTrans		   : COLOR0;
};



//-----------------------------------------------------------------------------
// Main
//-----------------------------------------------------------------------------
Conn main( Appdata In, 
		  uniform float4x4 modelview : register(VC_WORLD_PROJ),
		     uniform float3 ObjTrans : register(VC_OBJ_TRANS)

// could use VC_LIGHT_POS1,VC_EYE_POS (whatever effect your going for)
		  
		  )
{
   Conn Out;
   
   Out.TC0_Vertex_Position=mul(modelview,In.IN_Vertex_Position);
   Out.TC1_Vertex_Normal = In.IN_Vertex_Normal;
   Out.ObjTrans = float4(ObjTrans,1);
   Out.outTexCoord = In.texCoord;
   
   return Out;
}

toonP.hlsl

#define IN_HLSL
#include "shdrConsts.h"

const float4 Color1_ID9          = {0, 0, 0, 1}; //no comments
const float4 Color2_ID9          = {0, 0, 0, 1}; //no comments
const float4 Color3_ID9          = {0, 0, 0, 1}; //no comments


//Function (Block FID9)

float4 Toon_FID9(float4 Vertex_Position, float4 Vertex_Normal, float4 Light_Position, float4 Color1, float4 Color2, float4 Color3)
{
	float3 Look   = normalize(Light_Position.xyz - Vertex_Position.xyz);
	float3 Normal = normalize(Vertex_Normal.xyz);
	
	float Angle = acos( abs( dot( Normal, Look ) ) ) / 0.017453292;
	//change Color1 to black for outline
	Color1 = float4(0.3, 0, 0, 1);
	Color2 = float4(0.5, 0, 0, 1);
	Color3 = float4(0.9, 0, 0, 1);
	
	return (Angle > 60) ? Color1 : (Angle > 40) ? Color2 : Color3;
}


struct INPUT
{
	float4 TC0_Vertex_Position : TEXCOORD0; //no comments
	float2 texCoord        : TEXCOORD0;
	float4 TC1_Vertex_Normal   : TEXCOORD1; //no comments
	float4 ObjTrans			:COLOR0;
};


struct OUTPUT
{
	float4 Color0 : COLOR0; //no comments
};


//---------------------------------------------------------------------------------
// Main
//---------------------------------------------------------------------------------

OUTPUT main(INPUT In, uniform sampler2D diffuseMap      : register(S0))
{
	OUTPUT Out = (OUTPUT)0;


	//Block level'	

	float4 Vertex_Position_ID9 = In.TC0_Vertex_Position;
	float4 Vertex_Normal_ID9 = In.TC1_Vertex_Normal;

	float4 Color_ID9 = Toon_FID9(Vertex_Position_ID9, Vertex_Normal_ID9,In.ObjTrans, Color1_ID9, Color2_ID9, Color3_ID9);

	//Block level

	//Out.Color0 = Color_ID9;
	Out.Color0 = Color_ID9 * tex2D(diffuseMap, In.texCoord);
	
	return Out;
}

#8 I tried Danni's code on the regular orc, but all I got was a reddish tint. It's probably because I'm new at this. Any suggestions?

#9 Any Pictures of this working(not that I doubt) just want to see the coolness;)

#10 Bobby: Here it is: http://tinypic.com/view.php?pic=x2jhoz&s=4

EDIT: Whoops, you want a pic of it working, sorry, I misread your question...well, that pic is what I get when I try the code presented here, heh.

#11 Hey Jess,
did you ever get this working? i've added it and also only get the red tint.

#12 Timothy,

Technically I think I have it working. The red tint is coming from the fact that for some reason the shader had a red tint hardcoded.

In toonP.hlsl:

ColorID_9 = Toon_FID9(...);
Out.Color0 = ColorID_9 * tex2D(...);

So where does this ColorID_9 come from? Well, it's trying to shade the polygon into one of three shades (Since toon shading is all about quantization of what would otherwise be smooth shading. So let's take a look at the function Toon_FID9

float Angle = acos( abs( dot( Normal, Look ) ) ) / 0.017453292;

	//change Color1 to black for outline
	Color1 = float4(0.3, 0, 0, 1);
	Color2 = float4(0.5, 0, 0, 1);
	Color3 = float4(0.9, 0, 0, 1);
	
	return (Angle > 60) ? Color1 : (Angle > 40) ? Color2 : Color3;

Ok, so We have 3 colors Color1, Color2, and Color3, but what?! only the red is being adjusted. That's where the red tint is coming from. If you change it to be more even (shades of grey), then you'll get rid of the red tint and get dark shadows on the mesh, which is closer to what we wanted.

What I still don't understand is why the comment in the code says "change Color1 to black for outline", cause I didn't get an outline when I changed it.


So overall I'm still working on it, but I'm slowly understanding.

Some approaches to toon shading require multipasses for the outline, so check out the CelShade Outline thread to see our discussion on that.

#13 Im a total noob with shaders. Would anyone be able to say how to get colored lights in the scene to work with this shader?

#14 What do you mean?

#15 Well, I tried out the shader, and it looks great, but I tried putting an sgLightObject in the scene, and my colored light has no effect on the object I have this shader on.

#16 After a long night/morning of trial and error, and learning about shaders (which was cool, but wasn't the solution :) ), I figured out how to get the dynamic lighting working...

To use the default dynamic lighting, I just add the bold below in materials.cs CustomMaterial definition:

new CustomMaterial( ToonMaterialName )
{
        mapTo = "YourDiffuseTexture.png";
	texture[0] = "YourDiffuseTexture.png";
	shader = ToonShader;
	[b]dynamicLightingMaterial = AtlasDynamicLightingMaterial;
   	dynamicLightingMaskMaterial = AtlasDynamicLightingMaskMaterial;[/b]
	version = 2.0;
         
};

#17 Wow, I just stopped by to check out the new beta, browse the forums.. and i saw this thread and thought "uh oh.." i totally forgot i posted that crazy thing.. anyhow, yah the colors create a mask thats applied to the diffuse texture

see updated post

enjoy..
k

#18 Hey Kevin. Thanks for the awesome update! I was just about to try to do some more shader work but saw this great post.

You mentioned register substitution for sgLights. I ended investigation posting my last comment above for dynamic light materials because I could not find the proper register to work with sgLights. Do you happen to know which register this is in torque, maybe I overlooked it!

Thanks,
Dante

#19 Those are new to me, but i'd start looking at AtlasDynamicLightingMaterial
and see if its setting anything..

heh, looks like i got some catching up to do..

#20 Heh, yea that was part of what I spent a while doing (and failed lol). It also uses the light matrix, and passes that do a function to figure out the dynamic lighting. Couldn't find out a way to do this and keep the texture data, since im still a shader noob. Anyway, works fine with the above material properties I noted above, so I'm sticking with that for now :).

-Dante