cartoon outlining (cartoon rendering part1)
by F.W. Hardijzer · in Torque Game Engine · 05/31/2003 (3:22 pm) · 113 replies
Ok :)
I've been working on this for about a whole day, and still there are some bugs in it :(
But that doesn't hold me from posting it for others to use :)
so here is the code:
FOR SHAPES
in ts/tsMesh.cc
in TSMesh::render after:
FOR TERRAIN:
in terrain/terrRender.cc
in TerrainRender::renderXFCache after
FOR INTERIORS:
I'm still working on this one, and it's quite difficult (since there are lots of render codes in the interiors,
I've finally gotten some lines over it, but it is pretty glitchy, and will only look good on some interiors,
the $pref::renderOutline for interiors is $Pref::Interior::renderOutline,
and $pref::outlineWidth is $Pref::Interior::outlineWidth...
the reason i changed these vals for interiors is because i can imagine someone wanting terrain and shape, btu not this buggy interior code enabled :)
anyways, enough bullshit, code:
in interior/interiorRender.cc
in void Interior::render
after:
some prefs of this:
$Pref::renderOutline (true/false) render outlines or not
$Pref::outlineWidth (0-10) linewidth of outlines
You are free to use this code for whatever project you'd like, as long as you post every change you make on this forum so others can use it too :)
BUGS:
- enviroment mapped models have 'reflective' lines
Please if you like this stuff, contribute and help us with this!
also if you know anything about cell-shading, try to do that too...
in other words:
FRANK BIGNONE: please post your cell shading stuff from DOP here :(
and another word, noone is allowed to submit this as a resource except for me when i think the code is good enough to be posted as a resource :)
I've been working on this for about a whole day, and still there are some bugs in it :(
But that doesn't hold me from posting it for others to use :)
so here is the code:
FOR SHAPES
in ts/tsMesh.cc
in TSMesh::render after:
S32 drawType = getDrawType(draw.matIndex>>30);
glDrawElements(drawType,draw.numElements,GL_UNSIGNED_SHORT,&indices[draw.start]);
}add/*cell shading*/
bool doOutline=Con::getBoolVariable("$Pref::renderOutline",true);
if (doOutline) {
bool oldlighting=glIsEnabled(GL_LIGHTING);
glDisable(GL_LIGHTING);
F32 oldLineWidth;
int oldcullface;
glGetIntegerv(GL_CULL_FACE,&oldcullface);
glGetFloatv(GL_LINE_WIDTH,&oldLineWidth);
F32 lineWidth=Con::getFloatVariable("$Pref::OutlineWidth",2);
glLineWidth(lineWidth);
glCullFace(GL_FRONT);
glPolygonMode (GL_BACK, GL_LINE);
glDepthFunc(GL_LEQUAL);
bool old2dtex=glIsEnabled(GL_TEXTURE_2D);
glDisable(GL_TEXTURE_2D);
F32 oldcolor[4];
glGetFloatv(GL_CURRENT_COLOR,oldcolor);
glColor4f(0.0f,0.0f,0.0f,1.0f);
for (S32 i=0; i<primitives.size(); i++)
{
TSDrawPrimitive & draw = primitives[i];
AssertFatal(draw.matIndex & TSDrawPrimitive::Indexed,
"TSMesh::render: rendering of non-indexed meshes no longer supported");
S32 drawType = getDrawType(draw.matIndex>>30);
glDrawElements(drawType,draw.numElements,GL_UNSIGNED_SHORT,&indices[draw.start]);
}
glColor4fv(oldcolor);
if (old2dtex) glEnable(GL_TEXTURE_2D);
glDepthFunc(GL_LEQUAL);
glLineWidth(oldLineWidth);
glColor3f(1.0f,1.0f,1.0f);
glCullFace(GL_BACK);
glPolygonMode (GL_BACK, GL_FILL);
if (oldlighting) glEnable(GL_LIGHTING);
}
/* end cell shading*/FOR TERRAIN:
in terrain/terrRender.cc
in TerrainRender::renderXFCache after
U32 vertexCount = mXFIndexBuffer[count + 1];
glDrawElements(mode, vertexCount, GL_UNSIGNED_SHORT, mXFIndexBuffer + count + 2);add://Cell shading
/*cell shading*/
bool doOutline=Con::getBoolVariable("$Pref::renderOutline",true);
if (doOutline) {
/*extra cell shading*/
F32 oldLineWidth;
int oldcullface;
glGetIntegerv(GL_CULL_FACE,&oldcullface);
glGetFloatv(GL_LINE_WIDTH,&oldLineWidth);
F32 lineWidth=Con::getFloatVariable("$Pref::OutlineWidth",2);
glLineWidth(lineWidth);
//glBindTexture(GL_TEXTURE_2D,NULL);
glColor3f(0.0f,0.0f,0.0f);
glCullFace(GL_FRONT);
glPolygonMode (GL_BACK, GL_LINE);
//Draw here
glColor3f(0.0f,0.0f,0.0f);
glDisable(GL_TEXTURE_2D);
glDrawElements(mode, vertexCount, GL_UNSIGNED_SHORT, mXFIndexBuffer + count + 2);
glEnable(GL_TEXTURE_2D);
glLineWidth(oldLineWidth);
glCullFace(GL_BACK);
glPolygonMode (GL_BACK, GL_FILL);
/*end extra cs*/
}
/* end cell shading*/FOR INTERIORS:
I'm still working on this one, and it's quite difficult (since there are lots of render codes in the interiors,
I've finally gotten some lines over it, but it is pretty glitchy, and will only look good on some interiors,
the $pref::renderOutline for interiors is $Pref::Interior::renderOutline,
and $pref::outlineWidth is $Pref::Interior::outlineWidth...
the reason i changed these vals for interiors is because i can imagine someone wanting terrain and shape, btu not this buggy interior code enabled :)
anyways, enough bullshit, code:
in interior/interiorRender.cc
in void Interior::render
after:
if (smRenderMode != 0) {
PROFILE_START(IRO_DebugRender);
debugRender(pMaterials, instanceHandle);
PROFILE_END();
return;
}add://cartoon outline
bool doOutline=Con::getBoolVariable("$Pref::Interior::renderOutline",false);
if (doOutline) {
F32 oldLineWidth=1;
glGetFloatv(GL_LINE_WIDTH,&oldLineWidth);
F32 lineWidth=Con::getFloatVariable("$Pref::Interior::OutlineWidth",2);
glLineWidth(lineWidth);
// Base textures
glBlendFunc(GL_ONE, GL_ZERO);
glDisable(GL_TEXTURE_2D);
glColor3f(0, 0, 0);
for (U32 i = 0; i < sgActivePolyListSize; i++) {
const Surface& rSurface = mSurfaces[sgActivePolyList[i]];
glBegin(GL_LINE_LOOP);
glVertex3fv(mPoints[mWindings[rSurface.windingStart]].point);
S32 skip = rSurface.windingStart + 1;
while (skip < (rSurface.windingStart + rSurface.windingCount)) {
glVertex3fv(mPoints[mWindings[skip]].point);
skip += 2;
}
skip -= 1;
while (skip > rSurface.windingStart) {
if (skip < (rSurface.windingStart + rSurface.windingCount))
glVertex3fv(mPoints[mWindings[skip]].point);
skip -= 2;
}
glEnd();
}
glEnable(GL_TEXTURE_2D);
//return;
}
//End cartoon outlinesome prefs of this:
$Pref::renderOutline (true/false) render outlines or not
$Pref::outlineWidth (0-10) linewidth of outlines
You are free to use this code for whatever project you'd like, as long as you post every change you make on this forum so others can use it too :)
BUGS:
- enviroment mapped models have 'reflective' lines
Please if you like this stuff, contribute and help us with this!
also if you know anything about cell-shading, try to do that too...
in other words:
FRANK BIGNONE: please post your cell shading stuff from DOP here :(
and another word, noone is allowed to submit this as a resource except for me when i think the code is good enough to be posted as a resource :)
About the author
#42
02/16/2005 (11:07 am)
Nice resource though I think it's usually better to do this sort of thing in the modelling app.
#43
02/16/2005 (11:07 am)
Nice resource though I think it's usually better to do this sort of thing in the modelling app. Does it work with DX?
#44
- Add a renderOutline() sorta method to draw the mesh with backface cull and vertices offset by their normals
- Mess with the scenegraph so it'll call that renderOutline() on all shapes after the opaque geometry has been drawn
That way only the outlines will be really drawing, since the inner areas of the backface triangles will be Z-occluded already. It's also possible to draw all those shapes using only one GL state change (you can use the vertex colors to change the outline color per shape).
Also, you can get interesting effects if you draw the outlines *after* the transulscent objects have been drawn.
02/16/2005 (11:49 am)
The inverted faces can probably be done faster in real time, than encoding the faces with the moedels directly. Of course, this is much more work:- Add a renderOutline() sorta method to draw the mesh with backface cull and vertices offset by their normals
- Mess with the scenegraph so it'll call that renderOutline() on all shapes after the opaque geometry has been drawn
That way only the outlines will be really drawing, since the inner areas of the backface triangles will be Z-occluded already. It's also possible to draw all those shapes using only one GL state change (you can use the vertex colors to change the outline color per shape).
Also, you can get interesting effects if you draw the outlines *after* the transulscent objects have been drawn.
#45
02/16/2005 (1:49 pm)
@Colin haven't installed the fps gui yet...i'll post back later if it does kill my framerate. In experience though the game hasn't lagged up any for me...and that was using 1.3.0
#46
@Neto - not sure im following you all the way. It would be nice if the code could draw the polygon once, extrude verts along normals, flip the normals, color black, draw again. Although I don't understand how it would save framerate. Help me out with 'inner areas of the backface triangles will be z-occluded already'. Do you really think you could get an opaque outline around a transparent object? (Like a glass perhaps?)
02/16/2005 (2:50 pm)
Either method is essentially doubling the poly count no matter what you do. Although rendering the outline should take less time than the base model in both cases because you are dealing with lines or vertex colors, not textures. @Neto - not sure im following you all the way. It would be nice if the code could draw the polygon once, extrude verts along normals, flip the normals, color black, draw again. Although I don't understand how it would save framerate. Help me out with 'inner areas of the backface triangles will be z-occluded already'. Do you really think you could get an opaque outline around a transparent object? (Like a glass perhaps?)
#47
Toby.
03/06/2005 (9:33 am)
How are things moving along with this? It looks really nice and would be a wicked resource, we could expand it in many ways, specially with the scripting, so to call the outline on certain objects and not others, etc..Toby.
#48
Would you mind to share the files again? I'm interesting about the shading part instead of cartoon outline here.
Or someone got the code want to share would be great appreciated!
Steven
03/08/2005 (2:44 am)
@FRANKWould you mind to share the files again? I'm interesting about the shading part instead of cartoon outline here.
Or someone got the code want to share would be great appreciated!
Steven
#49
Reproducibility: Always
Platform: WinXp
Card: ATI 8500 All in Wonder
Drivers: Latest for OGL, DX, ATI.
Here is a list of the current bugs & Anomalies with Cel Shading that our team has found:
- Draws Lines of revery face under the terrain
- Doesn't take into account backfacing (similar prob to one above) -> Tree spites, etc...
- Huge differences in rendering on ATI drivers and Cards.
- Problems with lines in the inside of the models' outline (Like the above player model shows) -> Amplified with ATI cards/drivers differences
- Doesn't fully render terrain outline, as shown below.
- Creates weird rendering on the borders of the screen when within the terrain.
- It doesn't always take into account borders 'inside' an interior. So if you have a column on a plane, it might draw lines at the base of the column where it hits the other brush - when it shouldn't.

Toby.
03/08/2005 (7:08 am)
Engine: TGE 1.3Reproducibility: Always
Platform: WinXp
Card: ATI 8500 All in Wonder
Drivers: Latest for OGL, DX, ATI.
Here is a list of the current bugs & Anomalies with Cel Shading that our team has found:
- Draws Lines of revery face under the terrain
- Doesn't take into account backfacing (similar prob to one above) -> Tree spites, etc...
- Huge differences in rendering on ATI drivers and Cards.
- Problems with lines in the inside of the models' outline (Like the above player model shows) -> Amplified with ATI cards/drivers differences
- Doesn't fully render terrain outline, as shown below.
- Creates weird rendering on the borders of the screen when within the terrain.
- It doesn't always take into account borders 'inside' an interior. So if you have a column on a plane, it might draw lines at the base of the column where it hits the other brush - when it shouldn't.

Toby.
#50
Here is my revamped renderXFCache... Just replace the existing one with this one in terrain/terrRender.cc
void TerrainRender::renderXFCache()
{
U32 count = 0;
bool doOutline=Con::getBoolVariable("$Pref::renderOutline",true);
F32 oldLineWidth;
glGetFloatv(GL_LINE_WIDTH,&oldLineWidth);
F32 lineWidth=Con::getFloatVariable("$Pref::OutlineWidth",2);
while (count < mXFIndexCount)
{
U32 mode = mXFIndexBuffer[count];
U32 vertexCount = mXFIndexBuffer[count + 1];
// Cell Shading
if(doOutline)
{
glPolygonMode (GL_BACK, GL_LINE);
glLineWidth(lineWidth);
glDisable(GL_CULL_FACE);
glColor3f(0.0f,0.0f,0.0f);
glDisable(GL_TEXTURE_2D);
glDrawElements(mode, vertexCount, GL_UNSIGNED_SHORT, mXFIndexBuffer + count + 2);
glEnable(GL_TEXTURE_2D);
glEnable(GL_CULL_FACE);
glLineWidth(oldLineWidth);
glPolygonMode (GL_BACK, GL_FILL);
glDrawElements(mode, vertexCount, GL_UNSIGNED_SHORT, mXFIndexBuffer + count + 2);
}
// End cell shading
count += vertexCount + 2;
}
}
Comments appreciated.
03/09/2005 (7:42 pm)
The algorithm for terrain outlining doesn't work worth poop... So I rewrote it to make it work properly and optomized the heck out of it... (oh ya... if I can do this so can anyone, I don't know squat about OpenGL really) When I get around to actually cell shading everything, I will post it here. Should be by the end of the upcoming weekend.Here is my revamped renderXFCache... Just replace the existing one with this one in terrain/terrRender.cc
void TerrainRender::renderXFCache()
{
U32 count = 0;
bool doOutline=Con::getBoolVariable("$Pref::renderOutline",true);
F32 oldLineWidth;
glGetFloatv(GL_LINE_WIDTH,&oldLineWidth);
F32 lineWidth=Con::getFloatVariable("$Pref::OutlineWidth",2);
while (count < mXFIndexCount)
{
U32 mode = mXFIndexBuffer[count];
U32 vertexCount = mXFIndexBuffer[count + 1];
// Cell Shading
if(doOutline)
{
glPolygonMode (GL_BACK, GL_LINE);
glLineWidth(lineWidth);
glDisable(GL_CULL_FACE);
glColor3f(0.0f,0.0f,0.0f);
glDisable(GL_TEXTURE_2D);
glDrawElements(mode, vertexCount, GL_UNSIGNED_SHORT, mXFIndexBuffer + count + 2);
glEnable(GL_TEXTURE_2D);
glEnable(GL_CULL_FACE);
glLineWidth(oldLineWidth);
glPolygonMode (GL_BACK, GL_FILL);
glDrawElements(mode, vertexCount, GL_UNSIGNED_SHORT, mXFIndexBuffer + count + 2);
}
// End cell shading
count += vertexCount + 2;
}
}
Comments appreciated.
#51
Toby.
03/10/2005 (11:14 am)
Urm.. this makes the terrain invisible, which despite being interesting doesn't quite achieve the wanted goal.Toby.
#52
stick this in your prefs / call it from the console.
$pref::renderOutline = true;
oh ya... and delete prefs.cs.dso
03/10/2005 (1:23 pm)
This is exactly the code I am using, and it doesnt make the terrain invisible. It draws outlines around everything that dont mess up... stick this in your prefs / call it from the console.
$pref::renderOutline = true;
oh ya... and delete prefs.cs.dso
#53
I found the source of the bug - When you turn the rendering off, it makes the terrain invisible.
Aslo, your version had lead to some new bugs with black polys appearing all over the place.

Thanks,
Toby.
03/10/2005 (1:42 pm)
Chris,I found the source of the bug - When you turn the rendering off, it makes the terrain invisible.
Aslo, your version had lead to some new bugs with black polys appearing all over the place.

Thanks,
Toby.
#54
Ya, terrain invisiable was because you turned off $pref::renderOutline
03/10/2005 (2:12 pm)
There's no black polys anywhere on my screen... and Ive been running around the terrain for a good 6 hours... Where are you seeing black polys? Got a screenshot? or is that black poly on that screenie what you're talkign about?Ya, terrain invisiable was because you turned off $pref::renderOutline
#55
EDIT: If you have really bumpy terrain, it will outline a lot of the terrain, but it is supposed to do that.
03/10/2005 (2:15 pm)
All my code does is outlines all the back facing polys without clipping anything... The only thing it draws is the polys it would normally draw... So I don't see how it could draw a black poly. Unless you mean a black outlined poly.... In which case that could be... I had that problem but fixed it.. If you put the code the way I had it and add the pref you should render perfectly drawn outlines...EDIT: If you have really bumpy terrain, it will outline a lot of the terrain, but it is supposed to do that.
#56
I would presume this is a rendering difference between ATI and Nvidia, would I be right in saying you have an Nvidia card?
Toby.
03/10/2005 (4:23 pm)
Well as you can see from the screenshot this problem occurs on the terrain (black poly) and in mid air (line across the screen).I would presume this is a rendering difference between ATI and Nvidia, would I be right in saying you have an Nvidia card?
Toby.
#57
.. Can you figure out what part is giving you problems. It makes no sense that this shoudln't work. Try not disabling culling and setting it to cull front facing polys instead.
Does the original code work for you? Cause the only difference between it and mine is disabling culling...
03/10/2005 (5:23 pm)
I have an onboard NVidia card... However, I am just making OpenGL calls.. Nothing special... .. Can you figure out what part is giving you problems. It makes no sense that this shoudln't work. Try not disabling culling and setting it to cull front facing polys instead.
Does the original code work for you? Cause the only difference between it and mine is disabling culling...
#58
www.freewebs.com/ktchris/index.htm
03/10/2005 (5:37 pm)
While we'er on the topic of cell shading... I got my terrain cell shading code working... And, though there are lines, I have canyons, that basically look like there filled with paint. You can still walk in them, but because there are no shadows or textures or anything, you cant gauge depth... Do you put a light texturing on your solidly colored textures? Or do you use highliting from your light sources or what? I will write teh highliting code if i must... I will show you what it looks like on the website (gimme a couple minutes to get a screenshot up there)www.freewebs.com/ktchris/index.htm
#59
03/10/2005 (5:55 pm)
*is now stalking this thread*
#60
A comment I would make is instead of dark solid lines, if its possible,
maybe softer lines that make it look more hand-drawn or just less.. dark and outlined
more like accent lines
03/10/2005 (8:54 pm)
Wow, that effect looks really cool. Very Wind-waker-ey heheheA comment I would make is instead of dark solid lines, if its possible,
maybe softer lines that make it look more hand-drawn or just less.. dark and outlined
more like accent lines
Torque Owner Colin Millar
Has anyone else had issue with this code and framerate or is it just my build?