Deferred Shading
by Andrew Mac · in Torque 3D Professional · 06/22/2014 (9:38 am) · 116 replies
About 2 weeks ago Azaezel, Timmy and myself set out to bring Physical Based Shading to Torque. One of the biggest hurdles we ran into is the lighting information we need is not available at the stages we need it. This is due to Torque's prepass lighting (aka deferred lighting).
How does deferred lighting work?
Everything is rendered twice. Once to calculate lighting, and then everything is rendered again + the previously obtained lighting information for your final lit scene. This is why a 30k poly model loaded into Torque will show 60k.
This may sound very expensive but it's made in a way that the light prepass is as cheap as possible. It's by no means a bad method of doing things, nor was it a poor choice at the time. In fact, most engines around that time frame used deferred lighting. It also has the advantage of running better on older hardware.
How does deferred shading work?
Everything is rendered once, but it's done in a rather clever way. When rendering, a pixel shader is used to write all the data needed to different buffers. One for depth/normals, one for lighting (specular info, etc), and one for color. Then at the end a shader is used to combine all the information from the different buffers into a final picture.
This reduces rendering to once, but puts more stress on the graphics card and uses more video memory. This was not necessarily considered the better choice until recent years when graphics hardware has gotten better.
Why make the switch?
I hate to say "because the other guys are doing it" but if you look into it the big players that were using deferred lighting have made the change to deferred shading ( see: UE4 ). This will allow us to easily implement different lighting models like cook-torrance.
It also has the advantage of allowing some additional shaders and postfx that we couldn't really do before. Take for instance Lukas' SSGI shader. One big issue he ran into is the lack of an available albedo (or color) buffer to read just color data from. This is available with deferred shading.
What's the status?
We've been working on the conversion for about a week now and everything seems to be working quite well. There's a few missing features, and a handful of broken things but it's functional and working. Performance is about the same as before, which we consider to be a good thing since we haven't attempted any kind of optimization yet.
Conclusion
It's a pretty big change and will have a huge impact on materials, material shaders, and lighting. While we plan to continue working on this regardless for personal use, we would like to know if this is something the community is interested seeing make it's way into the main repo. This wouldn't be a 3.6 or 3.7 change, but further down the line (perhaps 4.0?).
Links
Link to the development repo:
github.com/andr3wmac/Torque3D/tree/deferred_shading
A guy's benchmarking of deferred lighting vs shading:
frictionalgames.blogspot.ca/2010/10/pre-pass-lighting-redux.html
An article that evaluates Deferred Lighting VS Deferred Shading:
gameangst.com/?p=141
How does deferred lighting work?
Everything is rendered twice. Once to calculate lighting, and then everything is rendered again + the previously obtained lighting information for your final lit scene. This is why a 30k poly model loaded into Torque will show 60k.
This may sound very expensive but it's made in a way that the light prepass is as cheap as possible. It's by no means a bad method of doing things, nor was it a poor choice at the time. In fact, most engines around that time frame used deferred lighting. It also has the advantage of running better on older hardware.
How does deferred shading work?
Everything is rendered once, but it's done in a rather clever way. When rendering, a pixel shader is used to write all the data needed to different buffers. One for depth/normals, one for lighting (specular info, etc), and one for color. Then at the end a shader is used to combine all the information from the different buffers into a final picture.
This reduces rendering to once, but puts more stress on the graphics card and uses more video memory. This was not necessarily considered the better choice until recent years when graphics hardware has gotten better.
Why make the switch?
I hate to say "because the other guys are doing it" but if you look into it the big players that were using deferred lighting have made the change to deferred shading ( see: UE4 ). This will allow us to easily implement different lighting models like cook-torrance.
It also has the advantage of allowing some additional shaders and postfx that we couldn't really do before. Take for instance Lukas' SSGI shader. One big issue he ran into is the lack of an available albedo (or color) buffer to read just color data from. This is available with deferred shading.
What's the status?
We've been working on the conversion for about a week now and everything seems to be working quite well. There's a few missing features, and a handful of broken things but it's functional and working. Performance is about the same as before, which we consider to be a good thing since we haven't attempted any kind of optimization yet.
Conclusion
It's a pretty big change and will have a huge impact on materials, material shaders, and lighting. While we plan to continue working on this regardless for personal use, we would like to know if this is something the community is interested seeing make it's way into the main repo. This wouldn't be a 3.6 or 3.7 change, but further down the line (perhaps 4.0?).
Links
Link to the development repo:
github.com/andr3wmac/Torque3D/tree/deferred_shading
A guy's benchmarking of deferred lighting vs shading:
frictionalgames.blogspot.ca/2010/10/pre-pass-lighting-redux.html
An article that evaluates Deferred Lighting VS Deferred Shading:
gameangst.com/?p=141
About the author
#42
On the cubemapping thing: There are a few commits in play already there rolling that subsystem on over to a more fully deferred pipeline, and taking advantage of Andrew packing the specular into the color render target. TODO list for full conversion at time of writing is correct the scattersky, and drill on down to where it's sampling the dynamic cubemap and correct the timing. Once that's in play, current thinking my end is decouple the cubemap reference from the material entry itsself, and shift that on over to say, levelinfo, or sun so that folks don't have to do a material per asset per level in order to get static reflectance. As a further goal, thinking cook up a brand new object for local-area cubes and do up some distance-approximation checks. (Third phase will likely end up fairly arcane and involve blending of some form. Still mulling that over, and could very well be over-thinking that. Wouldn't be the first time there.) If you want to jump in, by all means, we're not turning down helping hands.
06/29/2014 (8:29 pm)
@Felix: yes and no. It's fairly premature to talk about PBR before we've laid the groundwork for that, but using the disney/ue4 BRDF approach, you've pretty much got a binary mask for metal/non-metal and a greyscale roughness map (variant on specular gloss) that blends final color output with reflectance. www.youtube.com/watch?v=LP7HgIMv4Qo (Hour long vid going over the various systems, how they interact, and why, so pack a lunch. Definitely worth a watch though.)On the cubemapping thing: There are a few commits in play already there rolling that subsystem on over to a more fully deferred pipeline, and taking advantage of Andrew packing the specular into the color render target. TODO list for full conversion at time of writing is correct the scattersky, and drill on down to where it's sampling the dynamic cubemap and correct the timing. Once that's in play, current thinking my end is decouple the cubemap reference from the material entry itsself, and shift that on over to say, levelinfo, or sun so that folks don't have to do a material per asset per level in order to get static reflectance. As a further goal, thinking cook up a brand new object for local-area cubes and do up some distance-approximation checks. (Third phase will likely end up fairly arcane and involve blending of some form. Still mulling that over, and could very well be over-thinking that. Wouldn't be the first time there.) If you want to jump in, by all means, we're not turning down helping hands.
#43
Anyone who wants to grab the latest version and get testing than by all means do so, if you can code than get in there and help us out :-) :-)
06/30/2014 (3:39 am)
Agree with az, let's just keep this thread about deferred shading for now. We won't be moving on to PBS until the current work is complete and stable. Anyone who wants to grab the latest version and get testing than by all means do so, if you can code than get in there and help us out :-) :-)
#44
@Andrew If there's no huge cost to the 2nd option I would definitely go with that, as long as it's optional per-material. As you say most materials most likely will have white highlights.
@Azaezel Those upgrades sound very exciting. Just per-level specific cubemaps would be a huge change, not to mention having a cubemap entity that could be placed and updated ala Source engine.
On a slight sidenote, have you folks looked into cubemap mipmapping? Last I checked mips were not used the way Torque sets up cubemaps. Enabling this and in turn enabling the hlsl code for accessing mips would be a huge upgrade as well.
06/30/2014 (5:12 am)
Not saying that the thread should switch to a PBR discussion, I'm just wondering over the possibilities and limitations with this upgrade. That said I'm not a render programmer, and though I wish I could do more I can't really help out with much more than a 3d artist's input and some shader stuff...@Andrew If there's no huge cost to the 2nd option I would definitely go with that, as long as it's optional per-material. As you say most materials most likely will have white highlights.
@Azaezel Those upgrades sound very exciting. Just per-level specific cubemaps would be a huge change, not to mention having a cubemap entity that could be placed and updated ala Source engine.
On a slight sidenote, have you folks looked into cubemap mipmapping? Last I checked mips were not used the way Torque sets up cubemaps. Enabling this and in turn enabling the hlsl code for accessing mips would be a huge upgrade as well.
#45
The specular stuff i am working on has no noticeable effect on frame rate at all when rendering the terrain, so that is definitely a bonus. Comparing it against the default T3D i am seeing the exact same thing Andrew reports, without a lot of geometry the deferred shading is either equal or slightly slower than deferred lighting. When the geometry count starts climbing the benefits of the deferred shading start taking over and it is at the very worst equal to the default T3D, but in most cases out performs it (honestly these results i feel will change depending on GPU type).
I have most of the terrain sorted out now, it's only the macro map left to fix up, so hopefully not to far away and i can send a PR so andrew can add it and everyone can test and supply feedback
06/30/2014 (5:27 am)
Just an update on performance on my current setup (AMD 7850 2gb):The specular stuff i am working on has no noticeable effect on frame rate at all when rendering the terrain, so that is definitely a bonus. Comparing it against the default T3D i am seeing the exact same thing Andrew reports, without a lot of geometry the deferred shading is either equal or slightly slower than deferred lighting. When the geometry count starts climbing the benefits of the deferred shading start taking over and it is at the very worst equal to the default T3D, but in most cases out performs it (honestly these results i feel will change depending on GPU type).
I have most of the terrain sorted out now, it's only the macro map left to fix up, so hopefully not to far away and i can send a PR so andrew can add it and everyone can test and supply feedback
#46
I know a few people were asking for this feature for a while in the default T3D, this would give them a chance to try it with there own projects..Again thanks for all the hard work guys..Ya'll are awesome!!!!
06/30/2014 (6:38 am)
Could the specular work also be added to the default T3D deferred lighting build?I know a few people were asking for this feature for a while in the default T3D, this would give them a chance to try it with there own projects..Again thanks for all the hard work guys..Ya'll are awesome!!!!
#47
If someone was so inclined it doesn't seem difficult to add specular to terrains.
@ Everyone:
Update: specular power, specular strength, and gloss maps are now working.
In other news I've revamped glow a little bit. It's still functionally the same but it should be much faster. Instead of rendering objects again in a glow bin like the deferred lighting did, this is flagged in our material info buffer and can be processed by a slightly altered version of the glow shader.
What's the end result? You could enable glow on as many objects in a scene as you wish and there's no worry of performance impact beyond the cost of the shader itself. In my tests even having glow enabled on every object in the entire scene doesn't even have a notable performance impact:
i.imgur.com/G5Lh7kQ.jpg
I'm already hashing this out into a more flexible system that anyone can use to apply postfx shaders to specific materials. Stay tuned for more on that as it develops.
06/30/2014 (11:28 pm)
@ Kory: If someone was so inclined it doesn't seem difficult to add specular to terrains.
@ Everyone:
Update: specular power, specular strength, and gloss maps are now working.
In other news I've revamped glow a little bit. It's still functionally the same but it should be much faster. Instead of rendering objects again in a glow bin like the deferred lighting did, this is flagged in our material info buffer and can be processed by a slightly altered version of the glow shader.
What's the end result? You could enable glow on as many objects in a scene as you wish and there's no worry of performance impact beyond the cost of the shader itself. In my tests even having glow enabled on every object in the entire scene doesn't even have a notable performance impact:
i.imgur.com/G5Lh7kQ.jpg
I'm already hashing this out into a more flexible system that anyone can use to apply postfx shaders to specific materials. Stay tuned for more on that as it develops.
#50
Subsurface scattering requires the light information and the RGB scatter color. This uses up a lot of space in our GBuffer for the scatter color. So, I looked for alternative solutions. There's screen space subsurface scattering but I couldn't get it working very well. One solution I found sounds like it has additional benefits beyond just filling this void:
colinbarrebrisebois.com/2011/03/07/gdc-2011-approximating-translucency-for-a-fas...
Translucency mapping. How would it work? As far as I know the alpha channel of diffuse map is only used with transparency. You wouldn't use transparency and translucency at the same time so I'm thinking you put a translucency map into the alpha channel of the diffuse map and check off Translucent and you're all set.
I haven't done the mapping part yet, but I did port the hlsl code from the link above in to try it out, using a fixed thickness value for the whole object. Still needs some tweaks but here's the result:
@ Felix and other artists
Is translucency mapping an adequate replacement for subsurface scattering?
07/01/2014 (4:58 pm)
I fixed up HDR and Bloom today so all the original postFX are now fully working. The next item on my list is subsurface scattering.Subsurface scattering requires the light information and the RGB scatter color. This uses up a lot of space in our GBuffer for the scatter color. So, I looked for alternative solutions. There's screen space subsurface scattering but I couldn't get it working very well. One solution I found sounds like it has additional benefits beyond just filling this void:
colinbarrebrisebois.com/2011/03/07/gdc-2011-approximating-translucency-for-a-fas...
Translucency mapping. How would it work? As far as I know the alpha channel of diffuse map is only used with transparency. You wouldn't use transparency and translucency at the same time so I'm thinking you put a translucency map into the alpha channel of the diffuse map and check off Translucent and you're all set.
I haven't done the mapping part yet, but I did port the hlsl code from the link above in to try it out, using a fixed thickness value for the whole object. Still needs some tweaks but here's the result:
@ Felix and other artists
Is translucency mapping an adequate replacement for subsurface scattering?
#51
As hinted at above, there were a few alterations to how those work. They no longer require a second diffuse texture, just Diffuse, Cubemap, Specular map. Sans specmap, it'll default to full-bore reflection just like presently if you have no secondary diffuse to mask things.
07/01/2014 (9:10 pm)
Ok, cubemaps of all forms should be knocked out for review. List is: sykybox, scattersky, material-referenced-cubemaps (putting off the redirection of the shader constant till phase 2), waterplanes, and mats tagged with dynamic reflection (Also threw in dynamic reflection on tsStatics to scratch a "consistent application across all types of objects" itch).As hinted at above, there were a few alterations to how those work. They no longer require a second diffuse texture, just Diffuse, Cubemap, Specular map. Sans specmap, it'll default to full-bore reflection just like presently if you have no secondary diffuse to mask things.
#52
Used as Regular Specular Map:
i.imgur.com/JJ1t7N2.png
Used as Metalness Map:
i.imgur.com/US7enc0.png
Used as Metalness Map (increased spec strength):
i.imgur.com/jZnHebb.png
I don't have much of a reference to go off with this so if anyone has any objects with metalness maps they want to test the option is available in the latest build of testDS project in the development repo.
Here are the equations for metalness if anyone has any tweaks or suggestions:
07/01/2014 (9:29 pm)
One of the guys from Marmoset stopped by the IRC channel and was nice enough to give a run down on adding metalness as an alternative option in light of the removal of colored specular maps. I can't say sure for sure if I've got it in there correctly but here are some comparison shots:Used as Regular Specular Map:
i.imgur.com/JJ1t7N2.png
Used as Metalness Map:
i.imgur.com/US7enc0.png
Used as Metalness Map (increased spec strength):
i.imgur.com/jZnHebb.png
I don't have much of a reference to go off with this so if anyone has any objects with metalness maps they want to test the option is available in the latest build of testDS project in the development repo.
Here are the equations for metalness if anyone has any tweaks or suggestions:
Specular Map Color = 0.04 * ( 1 - Metalness ) + diffuseColor * Metalness Final Diffuse Color *= ( 1 - Metalness )
#54
07/02/2014 (10:15 am)
@Andrew, if we use diffuse map's alpha channel for translucency mapping... we can't use on foliage?
#55
07/02/2014 (7:51 pm)
@Andrew Hard to tell, I'd have to see it on a more organic surface like a head or something to say. One benefit of the current method is it requires no additional textures. I take it one would author a thickness map for this method?
#56
Here's a better demonstration:
I used this as a thickness map on the faces of the cube:
i.imgur.com/N3rYS62.png
There's a few variables in the equation that need tweaking. I don't like how circular the light shapes are, they should fall off as they spread out. I also need to add support for it to spot lights and vector lights. Beyond the cost of the loading the map, it doesn't take up much space in the gbuffer and it's calculated in the lighting pass so it's really cheap.
07/02/2014 (8:12 pm)
Yeah. To address the issue Luis raised I just separated it into a different map. Maybe I'll think of an optimization later. I'm just playing around with it now. Apparently you can generate them by inverting the AO map. Here's a better demonstration:
I used this as a thickness map on the faces of the cube:
i.imgur.com/N3rYS62.png
There's a few variables in the equation that need tweaking. I don't like how circular the light shapes are, they should fall off as they spread out. I also need to add support for it to spot lights and vector lights. Beyond the cost of the loading the map, it doesn't take up much space in the gbuffer and it's calculated in the lighting pass so it's really cheap.
#57
Timmy, since your working on Terrain, I don't know if this bug still exists, but here is a link. www.garagegames.com/community/forums/viewthread/135776
07/02/2014 (8:53 pm)
Andrew, You can do it by inverted AO in a pinch, but the best results are by creating a separate Transmission(Knald)/Thickness(xnormal)Map. Timmy, since your working on Terrain, I don't know if this bug still exists, but here is a link. www.garagegames.com/community/forums/viewthread/135776
#58
07/02/2014 (9:27 pm)
Can someone give me an object and proper transmission/thickness map that I can test this against? I've got sun and spot lights in there now.
#59
What is happening with the dxt5 issue, when the terrain cell material is put together it checks if the normal map has a dxt5 format and if so applies a special material feature called MFT_IsDXTnm (it's a swizzling trick). Cut a long story short, it assumes all dxt5 normal maps are actually dxt5nm format. Good explanation here tech-artists.org/wiki/Normal_map_compression.
*Edit:
If you wanted parallax support with dxt5nm it is certainly possible as there is actually two free channels left when using dxt5nm. I could certainly add this support in if people would like it.
07/02/2014 (10:13 pm)
@Andrew Robinson:What is happening with the dxt5 issue, when the terrain cell material is put together it checks if the normal map has a dxt5 format and if so applies a special material feature called MFT_IsDXTnm (it's a swizzling trick). Cut a long story short, it assumes all dxt5 normal maps are actually dxt5nm format. Good explanation here tech-artists.org/wiki/Normal_map_compression.
*Edit:
If you wanted parallax support with dxt5nm it is certainly possible as there is actually two free channels left when using dxt5nm. I could certainly add this support in if people would like it.
Torque Owner Lukas Joergensen
WinterLeaf Entertainment
As I understand it, materials for PBS are designed in a whole other way than normal materials, so you'd have to have 2 versions of each material to accomodate both pipelines.