Deferred Shading
by Andrew Mac · in Torque 3D Professional · 06/22/2014 (9:38 am) · 116 replies
About 2 weeks ago Azaezel, Timmy and myself set out to bring Physical Based Shading to Torque. One of the biggest hurdles we ran into is the lighting information we need is not available at the stages we need it. This is due to Torque's prepass lighting (aka deferred lighting).
How does deferred lighting work?
Everything is rendered twice. Once to calculate lighting, and then everything is rendered again + the previously obtained lighting information for your final lit scene. This is why a 30k poly model loaded into Torque will show 60k.
This may sound very expensive but it's made in a way that the light prepass is as cheap as possible. It's by no means a bad method of doing things, nor was it a poor choice at the time. In fact, most engines around that time frame used deferred lighting. It also has the advantage of running better on older hardware.
How does deferred shading work?
Everything is rendered once, but it's done in a rather clever way. When rendering, a pixel shader is used to write all the data needed to different buffers. One for depth/normals, one for lighting (specular info, etc), and one for color. Then at the end a shader is used to combine all the information from the different buffers into a final picture.
This reduces rendering to once, but puts more stress on the graphics card and uses more video memory. This was not necessarily considered the better choice until recent years when graphics hardware has gotten better.
Why make the switch?
I hate to say "because the other guys are doing it" but if you look into it the big players that were using deferred lighting have made the change to deferred shading ( see: UE4 ). This will allow us to easily implement different lighting models like cook-torrance.
It also has the advantage of allowing some additional shaders and postfx that we couldn't really do before. Take for instance Lukas' SSGI shader. One big issue he ran into is the lack of an available albedo (or color) buffer to read just color data from. This is available with deferred shading.
What's the status?
We've been working on the conversion for about a week now and everything seems to be working quite well. There's a few missing features, and a handful of broken things but it's functional and working. Performance is about the same as before, which we consider to be a good thing since we haven't attempted any kind of optimization yet.
Conclusion
It's a pretty big change and will have a huge impact on materials, material shaders, and lighting. While we plan to continue working on this regardless for personal use, we would like to know if this is something the community is interested seeing make it's way into the main repo. This wouldn't be a 3.6 or 3.7 change, but further down the line (perhaps 4.0?).
Links
Link to the development repo:
github.com/andr3wmac/Torque3D/tree/deferred_shading
A guy's benchmarking of deferred lighting vs shading:
frictionalgames.blogspot.ca/2010/10/pre-pass-lighting-redux.html
An article that evaluates Deferred Lighting VS Deferred Shading:
gameangst.com/?p=141
How does deferred lighting work?
Everything is rendered twice. Once to calculate lighting, and then everything is rendered again + the previously obtained lighting information for your final lit scene. This is why a 30k poly model loaded into Torque will show 60k.
This may sound very expensive but it's made in a way that the light prepass is as cheap as possible. It's by no means a bad method of doing things, nor was it a poor choice at the time. In fact, most engines around that time frame used deferred lighting. It also has the advantage of running better on older hardware.
How does deferred shading work?
Everything is rendered once, but it's done in a rather clever way. When rendering, a pixel shader is used to write all the data needed to different buffers. One for depth/normals, one for lighting (specular info, etc), and one for color. Then at the end a shader is used to combine all the information from the different buffers into a final picture.
This reduces rendering to once, but puts more stress on the graphics card and uses more video memory. This was not necessarily considered the better choice until recent years when graphics hardware has gotten better.
Why make the switch?
I hate to say "because the other guys are doing it" but if you look into it the big players that were using deferred lighting have made the change to deferred shading ( see: UE4 ). This will allow us to easily implement different lighting models like cook-torrance.
It also has the advantage of allowing some additional shaders and postfx that we couldn't really do before. Take for instance Lukas' SSGI shader. One big issue he ran into is the lack of an available albedo (or color) buffer to read just color data from. This is available with deferred shading.
What's the status?
We've been working on the conversion for about a week now and everything seems to be working quite well. There's a few missing features, and a handful of broken things but it's functional and working. Performance is about the same as before, which we consider to be a good thing since we haven't attempted any kind of optimization yet.
Conclusion
It's a pretty big change and will have a huge impact on materials, material shaders, and lighting. While we plan to continue working on this regardless for personal use, we would like to know if this is something the community is interested seeing make it's way into the main repo. This wouldn't be a 3.6 or 3.7 change, but further down the line (perhaps 4.0?).
Links
Link to the development repo:
github.com/andr3wmac/Torque3D/tree/deferred_shading
A guy's benchmarking of deferred lighting vs shading:
frictionalgames.blogspot.ca/2010/10/pre-pass-lighting-redux.html
An article that evaluates Deferred Lighting VS Deferred Shading:
gameangst.com/?p=141
About the author
#22
They cover how they do the rendering including a modified Deferred/Pre-pass system. I sounds pretty interesting and probably worth a read at least:
http://advances.realtimerendering.com/s2013/Tatarchuk-Destiny-SIGGRAPH2013.pdf
06/24/2014 (8:35 pm)
Another interesting paper from the engineers working on Destiny at Bungie.They cover how they do the rendering including a modified Deferred/Pre-pass system. I sounds pretty interesting and probably worth a read at least:
http://advances.realtimerendering.com/s2013/Tatarchuk-Destiny-SIGGRAPH2013.pdf
#23
Don't forget Timmy :P
@ Jeff Raab
That was an interesting read. I smiled about half way through because it's pretty much exactly what we're currently doing. Take a look at their modified deferred rendering system:
This is actually the result we came out with when we hacked the existing prepass lighting into deferred shading. We kept the lighting pass in tact. I didn't even know until now that we were really doing anything different than usual deferred shading. I guess the difference is rendering the lights without using multiple render targets for the output.
There's a few spots I'd like to quote from that document:
We'll have to look into these cheap decals. Both glow and decals are currently done with forward rendering after the deferred shading is done. These could be moved into the main render with a little work. I don't know if decals are really all that expensive, but I know we could save a render on glowing objects.
This is the same conclusion I'm seeing on all the write-ups on the topic from the past few years. Best I can tell there was a time when deferred lighting simply performed better on a larger range of hardware but with increased power of graphics hardware and increased scene complexity the double geometry pass is not really worth it's cost anymore.
We never really started this to increase speed, it's been more about making Torque look better. With that said though, I am noticing better framerates in complex scenes, with a bit of a dip in framerate on simple scenes. Seems there's a larger initial cost that pays off as you increase scene complexity.
06/24/2014 (9:20 pm)
@ KoryDon't forget Timmy :P
@ Jeff Raab
That was an interesting read. I smiled about half way through because it's pretty much exactly what we're currently doing. Take a look at their modified deferred rendering system:
This is actually the result we came out with when we hacked the existing prepass lighting into deferred shading. We kept the lighting pass in tact. I didn't even know until now that we were really doing anything different than usual deferred shading. I guess the difference is rendering the lights without using multiple render targets for the output.
There's a few spots I'd like to quote from that document:
Quote:
This final geometry pass is nice in that it allows custom shading per object. On the downside, the cost is a complete second pass over all geometry in the scene (and there are no exceptions to this, because we need some kind of surface color information!).
(Also, because you no longer have an albedo buffer anywhere, you lose the ability to have cheap decals. They can’t just modify that buffer.
Essentially they have to do their own lighting and blend on top of the Lit Result, like transparent geometry.)
We'll have to look into these cheap decals. Both glow and decals are currently done with forward rendering after the deferred shading is done. These could be moved into the main render with a little work. I don't know if decals are really all that expensive, but I know we could save a render on glowing objects.
Quote:
Overall, we like this approach. But, we really don’t want to pay for the full second pass over all the geometry.
From the standpoint of CPU setup cost and GPU vertex cost, full-screen shading is a clear win over a second geometry pass.
This is the same conclusion I'm seeing on all the write-ups on the topic from the past few years. Best I can tell there was a time when deferred lighting simply performed better on a larger range of hardware but with increased power of graphics hardware and increased scene complexity the double geometry pass is not really worth it's cost anymore.
We never really started this to increase speed, it's been more about making Torque look better. With that said though, I am noticing better framerates in complex scenes, with a bit of a dip in framerate on simple scenes. Seems there's a larger initial cost that pays off as you increase scene complexity.
#24
06/25/2014 (6:53 am)
Decals are another transparency blend - they can really add up. And Torque decals can get pretty complex; shoot a rocket at the tower asset in a corner somewhere and you'll see what I'm talking about. But it sounds like they were happy with the trade-off - just generate the decal geometry and pass it in with the rest of the scene. 
#25
06/25/2014 (5:48 pm)
Oh sorry, thanks Timmy and anyone else helping with this ya'll are awesome!!!
#26
Deferred Lighting (3.5.1):
i.imgur.com/BYDNOh1.jpg
Deferred Shading:
i.imgur.com/xETsVTU.jpg
There's a few missing features which are evident in the shots like alpha threshold and emissive but we're working on it.
I should mention, I've done away with per-material specular highlight colors. At best it can be a property of the light but it's not feasible to use a per-material RGB specular color. What I've done instead is greyscale the spec map and pack it into a single channel. It seems to work quite well. The majority of my spec maps are greyscale anyway. Gloss map in alpha will still work for spec power when it's implemented.
*edit: Updated stock screenshot. The one I had before was running abnormally slow for some reason.
06/26/2014 (7:17 pm)
I've made a lot of progress on lighting and specular. It's lacking in spec power and spec strength controls but the actual blinn-phong specular calculations are in place. Here's a comparison shot from good old chinatown. This is all on the repo now so you if you want to check it out please do. Just remember it's all in the testDS project in My Projects, templates are not up to date and will crash. Deferred Lighting (3.5.1):
i.imgur.com/BYDNOh1.jpg
Deferred Shading:
i.imgur.com/xETsVTU.jpg
There's a few missing features which are evident in the shots like alpha threshold and emissive but we're working on it.
I should mention, I've done away with per-material specular highlight colors. At best it can be a property of the light but it's not feasible to use a per-material RGB specular color. What I've done instead is greyscale the spec map and pack it into a single channel. It seems to work quite well. The majority of my spec maps are greyscale anyway. Gloss map in alpha will still work for spec power when it's implemented.
*edit: Updated stock screenshot. The one I had before was running abnormally slow for some reason.
#27
Spec disabled(Default T3D):
i.imgur.com/XYUEBh0.png
Spec enabled:
i.imgur.com/9X8puzu.png
This hasn't been uploaded yet as i'm still working on it.
06/27/2014 (10:15 pm)
Using deferred shading also allows us to implement a very cheap method of using grey scale specular maps with the terrain layers.Spec disabled(Default T3D):
i.imgur.com/XYUEBh0.png
Spec enabled:
i.imgur.com/9X8puzu.png
This hasn't been uploaded yet as i'm still working on it.
#28
06/27/2014 (11:06 pm)
Really impressive work guys. 
#30
06/28/2014 (4:43 am)
the spec enabled looks great
#31
06/28/2014 (5:35 am)
With the spec map, it is tied into the detail texture and uses the alpha channel. With the current setup in deferred shading we have to essentially write 0's to the alpha channel of the color target(specular) in the gbuffer when redering the terrain, packing a spec map into the alpha channel of the detail map comes very cheap because we are sampling the detail texture regardless and instead of writing 0's we can output out the spec map value instead. It's pretty much a win-win scenario.
#32
But thought you might as well learn of some of the issues I had with that approach.
Alpha values close to 0 makes the detail texture excluded, so unless you fix that, you have to make it a must that there is never any alpha values below some value close to 0 (didn't find the exact value that excludes it).
Also I read a discussion about the detail alpha channel where (I believe it was Tom Spilman) said that he was experimenting with using the alpha channel for adjusting the blending pattern, and I'm not sure if it ever made it into stock.
06/28/2014 (5:40 am)
@Timmy sounds like it was a good idea of me to stop using the detail maps alpha channel for blending then :PBut thought you might as well learn of some of the issues I had with that approach.
Alpha values close to 0 makes the detail texture excluded, so unless you fix that, you have to make it a must that there is never any alpha values below some value close to 0 (didn't find the exact value that excludes it).
Also I read a discussion about the detail alpha channel where (I believe it was Tom Spilman) said that he was experimenting with using the alpha channel for adjusting the blending pattern, and I'm not sure if it ever made it into stock.
#33
I'm just playing with ideas and trying to improve the current setup is all. With every new feature there will always be pros and cons. It can be rather difficult to implement something new and keep everyone else working 100%. I'm very open to ideas.....
06/28/2014 (5:42 am)
Well we can't keep everyone happy lol. There is also the option of using the dxt5 swizzle trick on the normal map and this leaves two channels free with that texture, one for height (parallax) and one for spec. I'm not a huge fan of tying the spec in with the normal map though, but it could be made to work.I'm just playing with ideas and trying to improve the current setup is all. With every new feature there will always be pros and cons. It can be rather difficult to implement something new and keep everyone else working 100%. I'm very open to ideas.....
#34
06/28/2014 (5:45 am)
That would make me sad :( I use high-pass textures for my detail textures and not gray scales, I find that high-pass textures gives a better result.
#35
06/28/2014 (5:51 am)
Oops edited my last post instead of posting a new one, refer to that one for reply
#36
I can't imagine it'd be that hard, alternatively just set some minimum alpha value on the detail map alpha. I'd rather have that than remove a rather pleasant feature :)
06/28/2014 (6:10 am)
Hmm I think it'd be better to try and solve the issues with the detail map alpha than try to avoid it, to dodge the issues.I can't imagine it'd be that hard, alternatively just set some minimum alpha value on the detail map alpha. I'd rather have that than remove a rather pleasant feature :)
#37
If i get time tomorrow(or early next week) i'll finish it off and send a PR to andrew and than people can test it and offer some thoughts/opinions etc on any problems or whatever with it.
06/28/2014 (6:18 am)
Yeah the alpha problem with the detail map is very minor, i can already think of a good work around for that so all good :-)If i get time tomorrow(or early next week) i'll finish it off and send a PR to andrew and than people can test it and offer some thoughts/opinions etc on any problems or whatever with it.
#38
The problem you guys are experiencing putting values into alpha is because of the blend mode setup for the stateblock for the shaders. I ran into this issue before. What's happening is there's a blend mode setup that causing it to perform an operation when storing the new values. I found this in terCellMaterial:
You could always change it to a straight copy and alter the shaders to do the alpha blending manually. This will allow you to overwrite alpha channel without issue.
06/28/2014 (8:06 am)
Quote:
Alpha values close to 0 makes the detail texture excluded, so unless you fix that, you have to make it a must that there is never any alpha values below some value close to 0 (didn't find the exact value that excludes it).
The problem you guys are experiencing putting values into alpha is because of the blend mode setup for the stateblock for the shaders. I ran into this issue before. What's happening is there's a blend mode setup that causing it to perform an operation when storing the new values. I found this in terCellMaterial:
desc.setBlend( true, GFXBlendSrcAlpha, GFXBlendInvSrcAlpha );
You could always change it to a straight copy and alter the shaders to do the alpha blending manually. This will allow you to overwrite alpha channel without issue.
#39
06/29/2014 (12:05 pm)
@Andrew When you say you've done away with specular color entirely, do you mean only allowing greyscale specular maps or completely removing any coloration of the spec highlights? Although not used to much effect right now this would be really important if Torque ever moves to PBR and/or starts offering some better cubemap reflection options (such as baking ones in levels).
#40
Unfortunately, what I'm running into is that it's too much cost to use a specular color at all. I can do per-material specular power, and specular strength, but color takes up a lot of space. I'm looking at two options (unless someone has another idea).
1) Pure white specular highlights. It seems like this works in the majority of situations. Screenshot: i.imgur.com/XC2glWQ.jpg
2) Multiply the specular highlights by the diffuse light color before application. I read something about Unity faking specular colors by doing this:
i.imgur.com/LNfSF8U.jpg
What option should I go with?
06/29/2014 (4:34 pm)
The way I've been looking at it is physical based shading will be a graphics option. It's going to have a significant performance impact so it will have to be a something that's switched on with the highest level of graphics settings. When this occurs we could easily introduce another buffer if it's required. So, for the time being I'm just trying to restore as many of torque's existing features as I can so when PBS is switched off you'll have a regular functioning blinn-phong.Unfortunately, what I'm running into is that it's too much cost to use a specular color at all. I can do per-material specular power, and specular strength, but color takes up a lot of space. I'm looking at two options (unless someone has another idea).
1) Pure white specular highlights. It seems like this works in the majority of situations. Screenshot: i.imgur.com/XC2glWQ.jpg
2) Multiply the specular highlights by the diffuse light color before application. I read something about Unity faking specular colors by doing this:
i.imgur.com/LNfSF8U.jpg
What option should I go with?
Torque Owner Kory Imaginism
innovative imaginations
Keep up the great Research and work you guys are doing!!