Cell Shading Tutorial
by Arland (Barry) Woodham · in Game Design and Creative Issues · 01/06/2005 (8:29 am) · 22 replies
***Warning this is not real cell shading, but instead a cartoon outlining.***
Well I promised I would make this tutorial eventually and now I am just finally getting around to do so.
This is a tutorial that I am creating based on an idea I saw on a website sometime ago that for the time being escapes me. What the person had done was to create a sort of pseudo cell shading effect in Quake II. When you see how this technique works you will understand why the poly limits of this engine made it difficult. While helping a friend in class with his Unreal level that had a cartoony look I decided on a whim to try and implement the technique. The results of this were better than I hoped and so I went on to attempt it in Torque which took some more work but the result was worth it. Anyways enough talk, on to the actual tutorial.
Programs Used
Wings 3D
Photoshop
Milkshape
Torque
You may substitute your favorite applications but be aware some of the techniques may be slightly different in them.
Part One The Modeling
The first step is to model the object that you are going to cell-shade. In my case this is just an extruded primitive with a smooth modifier.

Next you need to copy you model. In wings this is done by selecting the whole model and then right clicking then clicking duplicate. Make sure the copy is in the same spot as the original.
Now you want to enlarge the model based on it's normals. For wings select all the faces and then use the bump command from the right click menu. In 3dsmax the modifier to use is called push. I am sorry but I do not know of what commands would be used in other programs.
What you want is a slightly larger version of your model as shown here. The copy is the wire frame. This is what will create you outline so the father it is from the base model the thicker the line will be.

The last step as far as the modeling goes is to invert the normals on the copy. In wings this is done by using invert after selecting the whole object. In max select all the faces and then use the flip command.

Now that all your modeling is done setup whatever uv's you want to do and export the file for milkshape. This is for those following along in wings3d not other modeling software.
Part Two Texturing
This is the easy part. If you have a fancy model you can spend quite some time here but for the sake of this tutorial I will stick to solid colors with no uv's.
When you look at the object in milkshape it will look something like this don't worry just right click and turn off draw backfaces.

First you want to apply your texture to the first object. This is what the object will look like so it is where you want to spend the most time. For my sake I am going to throw a solid color on. After that apply a solid black texture to the copy, the inverted object. And since the black is a solid color and will be used again and again you can make one small texture file and reuse it for all the models.
When you are done your model will look something like this and you will already be able to see how the shading effect works.

Part Three Torque
Of course the effect works well in the modeling program but we want it in torque after all so fire up your dts exporter and export it. Then copy your dts and your two textures to torque/starter.racing/data/shapes or whatever folder you need it in for you particular project.
All that's left is to go into the mission editor and add you object and admire its wonderful cartoon outlining.

Well I promised I would make this tutorial eventually and now I am just finally getting around to do so.
This is a tutorial that I am creating based on an idea I saw on a website sometime ago that for the time being escapes me. What the person had done was to create a sort of pseudo cell shading effect in Quake II. When you see how this technique works you will understand why the poly limits of this engine made it difficult. While helping a friend in class with his Unreal level that had a cartoony look I decided on a whim to try and implement the technique. The results of this were better than I hoped and so I went on to attempt it in Torque which took some more work but the result was worth it. Anyways enough talk, on to the actual tutorial.
Programs Used
Wings 3D
Photoshop
Milkshape
Torque
You may substitute your favorite applications but be aware some of the techniques may be slightly different in them.
Part One The Modeling
The first step is to model the object that you are going to cell-shade. In my case this is just an extruded primitive with a smooth modifier.

Next you need to copy you model. In wings this is done by selecting the whole model and then right clicking then clicking duplicate. Make sure the copy is in the same spot as the original.
Now you want to enlarge the model based on it's normals. For wings select all the faces and then use the bump command from the right click menu. In 3dsmax the modifier to use is called push. I am sorry but I do not know of what commands would be used in other programs.
What you want is a slightly larger version of your model as shown here. The copy is the wire frame. This is what will create you outline so the father it is from the base model the thicker the line will be.

The last step as far as the modeling goes is to invert the normals on the copy. In wings this is done by using invert after selecting the whole object. In max select all the faces and then use the flip command.

Now that all your modeling is done setup whatever uv's you want to do and export the file for milkshape. This is for those following along in wings3d not other modeling software.
Part Two Texturing
This is the easy part. If you have a fancy model you can spend quite some time here but for the sake of this tutorial I will stick to solid colors with no uv's.
When you look at the object in milkshape it will look something like this don't worry just right click and turn off draw backfaces.

First you want to apply your texture to the first object. This is what the object will look like so it is where you want to spend the most time. For my sake I am going to throw a solid color on. After that apply a solid black texture to the copy, the inverted object. And since the black is a solid color and will be used again and again you can make one small texture file and reuse it for all the models.
When you are done your model will look something like this and you will already be able to see how the shading effect works.

Part Three Torque
Of course the effect works well in the modeling program but we want it in torque after all so fire up your dts exporter and export it. Then copy your dts and your two textures to torque/starter.racing/data/shapes or whatever folder you need it in for you particular project.
All that's left is to go into the mission editor and add you object and admire its wonderful cartoon outlining.

About the author
#2
Gary (-;
01/06/2005 (10:09 am)
You know, that's a disturbingly elegant and simple way to do that. I'm VERY impressed with your method.Gary (-;
#4
niiice......i'd been looking for this in a few places, but couldn't find it...neato implementation!
on a side question (coming from a newbie) - how would this affect the polycount?
01/06/2005 (8:43 pm)
YAYAYYAYAYAYYAYYYY!!!!!!!!niiice......i'd been looking for this in a few places, but couldn't find it...neato implementation!
on a side question (coming from a newbie) - how would this affect the polycount?
#5
I have done this before in code...
Basically its the same thing, but instead of making an entire outer shell of the model... you detect the edges dynamically in-game and extrude the edge (each vertex extruded on its normal according to how thick you want the line)... and then you make it black or whatever color you want.
The above mentioned method is problematic when it comes to animations too... and overall effeciency, but for simple static shapes, it works fine. I've tried it that way before... but finally I got it to work in code instead (using DirectX).
01/06/2005 (9:08 pm)
As he said, it doubles your poly count.I have done this before in code...
Basically its the same thing, but instead of making an entire outer shell of the model... you detect the edges dynamically in-game and extrude the edge (each vertex extruded on its normal according to how thick you want the line)... and then you make it black or whatever color you want.
The above mentioned method is problematic when it comes to animations too... and overall effeciency, but for simple static shapes, it works fine. I've tried it that way before... but finally I got it to work in code instead (using DirectX).
#6
01/09/2005 (7:25 pm)
This is going in my favorites.
#7
Also, it might be a good idea to MultiResing the outline mesh. You don't need horrendous quality for an outline afterall, AND you might even be able to obtain a more artistic jagged outline look for the character.
01/11/2005 (3:48 am)
I think it would be better to make an addendum to the rendering code... Its not a very efficient way to do it, AND it is difficult to apply uniformly to all models. Still though, its a pretty decent temp fix.Also, it might be a good idea to MultiResing the outline mesh. You don't need horrendous quality for an outline afterall, AND you might even be able to obtain a more artistic jagged outline look for the character.
#8
Programs Used
Wings 3D
Photoshop
Milkshape
Torque
I know you said that others could be used, and I'd already be using torque and photoshop, but instead of milkshape and wings3d, could I substitute just 3d studio max? because I'm not sure what wings3d is. I was just wondering if I could do the steps you use wings3d for in 3d studio max. Thanks :)
05/13/2005 (5:42 am)
I've been trying to keep up/learn your cell shading / cartooning technique although havent gotten time to actually sit down and start trying it yet, and I was wondering if ALL of the following were all needed:Programs Used
Wings 3D
Photoshop
Milkshape
Torque
I know you said that others could be used, and I'd already be using torque and photoshop, but instead of milkshape and wings3d, could I substitute just 3d studio max? because I'm not sure what wings3d is. I was just wondering if I could do the steps you use wings3d for in 3d studio max. Thanks :)
#9
05/13/2005 (5:45 pm)
Yeah go ahead and use max. We used it for all the cartoon combat stuff only real difference is use the push modifier as I mentioned in the tutorial thats about the only difference there should be.
#10
01/02/2006 (2:55 pm)
Thanks for that "cartoon outlining" ... works perfekt :-)
#11
I got the idea from this thread and have tried it, just not to create an outlined object:
www.garagegames.com/mg/forums/result.thread.php?qt=38299
01/05/2006 (7:30 pm)
Couldn't you use the trick to bake it into the object using procedural textures? That way it just becomes a neat way to create textures. I got the idea from this thread and have tried it, just not to create an outlined object:
www.garagegames.com/mg/forums/result.thread.php?qt=38299
#12
01/14/2006 (3:27 pm)
Doh, I am an idiot. The cel shading comes from the seeing the backside of a normal flipped polygon. Very clever. my idea of baking it in place will not work.
#13
i tried it with 3dsmax and i works well.
the outline looks much better than drawing edges for outline.
instead of making model with scaled fliped normals,
i want to modify engine to draw mesh with scaled and flipped normals.
void TSMesh::render(S32 frame, S32 matFrame, TSMaterialList * materials, const MatrixF *objectTransform)
{
// draw normals
// draw scaled flipped normals with black color
}
i can draw flipped normals with glCullFace(GL_FRONT), right?
how can i scale it?
do i have to scale each vertices? how can i do that?
i'm a 3d programming newbie. need help!!
thanks
07/23/2006 (6:23 pm)
Thanks for great tip!!i tried it with 3dsmax and i works well.
the outline looks much better than drawing edges for outline.
instead of making model with scaled fliped normals,
i want to modify engine to draw mesh with scaled and flipped normals.
void TSMesh::render(S32 frame, S32 matFrame, TSMaterialList * materials, const MatrixF *objectTransform)
{
// draw normals
// draw scaled flipped normals with black color
}
i can draw flipped normals with glCullFace(GL_FRONT), right?
how can i scale it?
do i have to scale each vertices? how can i do that?
i'm a 3d programming newbie. need help!!
thanks
#14
01/18/2007 (5:27 am)
Ware is the exporter and export? PLEASE HELP!!!!
#15
11/25/2007 (1:06 am)
Great thread. how about people posting models they have made using this technique to showcase?
#16
02/28/2009 (11:55 am)
Sorry to bring up a 2 year old thread. However, one guy stated it can be problematic if the shapes are animated. I need a cell shaded player. Will this not work with any animated shapes at all? What's the best way to cell shade animated objects, then?
#17
02/28/2009 (12:22 pm)
Are you using TGEA? If so, cel-shading shaders are available, I *think*. If you're using TGE, you could add the MK and Koushik's code to enable DTS shaders.
#18
Where can I find these? Will they work with animations, too?
Thanks.
02/28/2009 (1:05 pm)
I'm using TGE 1.4.2.Quote:
you could add the MK and Koushik's code to enable DTS shaders.
Where can I find these? Will they work with animations, too?
Thanks.
#19
02/28/2009 (2:23 pm)
@xxShanexx09: the code for the Modernization Kit is in this private TGE thread, and Koushiks's code, TGE Shader Pack, is a resource.
#20
1) I feel like I'm missing out. Is there a version for the Torque Game Engine 1.4.2?
2) Koushik's resource is in a '.rar' file. How do I open those up? I'm on a Mac.
Thanks.
02/28/2009 (2:31 pm)
Wow! The modernization kit just looks too d*mn good according to the screen shot with the elf. Quote:
This is for TGE 1.5.0/1.5.1 ONLY!
1) I feel like I'm missing out. Is there a version for the Torque Game Engine 1.4.2?
2) Koushik's resource is in a '.rar' file. How do I open those up? I'm on a Mac.
Thanks.
Torque Owner Arland (Barry) Woodham
Well not really I just wanted to leave a few parting comments. The technique works best on simpler objects and works by seeing through the backs of polygons to expose the model inside. Therefore it does not create lines on the vehicle but instead a solid black mass which is "masked" by the vehicle and as such many interior lines will only show up as "shaded" if viewed from certain angles. Also be aware that this technique double the amount of polys that you are using for each object so model for half the poly count that you would use in a realistic game. Hope you have enjoyed reading this and some one will get some use out of it.