Deferred Shading
by Andrew Mac · in Torque 3D Professional · 06/22/2014 (9:38 am) · 116 replies
About 2 weeks ago Azaezel, Timmy and myself set out to bring Physical Based Shading to Torque. One of the biggest hurdles we ran into is the lighting information we need is not available at the stages we need it. This is due to Torque's prepass lighting (aka deferred lighting).
How does deferred lighting work?
Everything is rendered twice. Once to calculate lighting, and then everything is rendered again + the previously obtained lighting information for your final lit scene. This is why a 30k poly model loaded into Torque will show 60k.
This may sound very expensive but it's made in a way that the light prepass is as cheap as possible. It's by no means a bad method of doing things, nor was it a poor choice at the time. In fact, most engines around that time frame used deferred lighting. It also has the advantage of running better on older hardware.
How does deferred shading work?
Everything is rendered once, but it's done in a rather clever way. When rendering, a pixel shader is used to write all the data needed to different buffers. One for depth/normals, one for lighting (specular info, etc), and one for color. Then at the end a shader is used to combine all the information from the different buffers into a final picture.
This reduces rendering to once, but puts more stress on the graphics card and uses more video memory. This was not necessarily considered the better choice until recent years when graphics hardware has gotten better.
Why make the switch?
I hate to say "because the other guys are doing it" but if you look into it the big players that were using deferred lighting have made the change to deferred shading ( see: UE4 ). This will allow us to easily implement different lighting models like cook-torrance.
It also has the advantage of allowing some additional shaders and postfx that we couldn't really do before. Take for instance Lukas' SSGI shader. One big issue he ran into is the lack of an available albedo (or color) buffer to read just color data from. This is available with deferred shading.
What's the status?
We've been working on the conversion for about a week now and everything seems to be working quite well. There's a few missing features, and a handful of broken things but it's functional and working. Performance is about the same as before, which we consider to be a good thing since we haven't attempted any kind of optimization yet.
Conclusion
It's a pretty big change and will have a huge impact on materials, material shaders, and lighting. While we plan to continue working on this regardless for personal use, we would like to know if this is something the community is interested seeing make it's way into the main repo. This wouldn't be a 3.6 or 3.7 change, but further down the line (perhaps 4.0?).
Links
Link to the development repo:
github.com/andr3wmac/Torque3D/tree/deferred_shading
A guy's benchmarking of deferred lighting vs shading:
frictionalgames.blogspot.ca/2010/10/pre-pass-lighting-redux.html
An article that evaluates Deferred Lighting VS Deferred Shading:
gameangst.com/?p=141
How does deferred lighting work?
Everything is rendered twice. Once to calculate lighting, and then everything is rendered again + the previously obtained lighting information for your final lit scene. This is why a 30k poly model loaded into Torque will show 60k.
This may sound very expensive but it's made in a way that the light prepass is as cheap as possible. It's by no means a bad method of doing things, nor was it a poor choice at the time. In fact, most engines around that time frame used deferred lighting. It also has the advantage of running better on older hardware.
How does deferred shading work?
Everything is rendered once, but it's done in a rather clever way. When rendering, a pixel shader is used to write all the data needed to different buffers. One for depth/normals, one for lighting (specular info, etc), and one for color. Then at the end a shader is used to combine all the information from the different buffers into a final picture.
This reduces rendering to once, but puts more stress on the graphics card and uses more video memory. This was not necessarily considered the better choice until recent years when graphics hardware has gotten better.
Why make the switch?
I hate to say "because the other guys are doing it" but if you look into it the big players that were using deferred lighting have made the change to deferred shading ( see: UE4 ). This will allow us to easily implement different lighting models like cook-torrance.
It also has the advantage of allowing some additional shaders and postfx that we couldn't really do before. Take for instance Lukas' SSGI shader. One big issue he ran into is the lack of an available albedo (or color) buffer to read just color data from. This is available with deferred shading.
What's the status?
We've been working on the conversion for about a week now and everything seems to be working quite well. There's a few missing features, and a handful of broken things but it's functional and working. Performance is about the same as before, which we consider to be a good thing since we haven't attempted any kind of optimization yet.
Conclusion
It's a pretty big change and will have a huge impact on materials, material shaders, and lighting. While we plan to continue working on this regardless for personal use, we would like to know if this is something the community is interested seeing make it's way into the main repo. This wouldn't be a 3.6 or 3.7 change, but further down the line (perhaps 4.0?).
Links
Link to the development repo:
github.com/andr3wmac/Torque3D/tree/deferred_shading
A guy's benchmarking of deferred lighting vs shading:
frictionalgames.blogspot.ca/2010/10/pre-pass-lighting-redux.html
An article that evaluates Deferred Lighting VS Deferred Shading:
gameangst.com/?p=141
About the author
#2
06/22/2014 (11:31 am)
It doesn't differ on the actual rendering level. No alterations were made to the GFX layer. There should be no difference in OpenGL vs DirectX except that we'll have to write GLSL versions of all the shaders we've made ( not difficult ). There might be some performance impacts (positive or negative) but someone more versed in OpenGL will have to weigh in on that. 
#3
I'm going to have lots of fun porting this to OpenGL.
Thx for your work.
06/22/2014 (12:24 pm)
Hey guys, great project :DI'm going to have lots of fun porting this to OpenGL.
Thx for your work.
#5
Also another advantage with deferred shading is (in theory anyway) it's less taxing on the cpu because you are not doing that second geometry pass. Again this may not lead to any FPS increase at all but using less cpu can never be a bad thing. I guess when the project is a little more advanced some detailed profiling and benchmarking will be in order.
06/22/2014 (4:22 pm)
Another example of a big studio changing over to deferred shading is CryTek . They changed to deferred shading when creating RUSE because of PBS too. Talking of CryTek this doesn't sound very good for them (sorry off-topic):tech4gamers.com/crytek-may-go-bankrupt-owing-to-ruse-failure/Also another advantage with deferred shading is (in theory anyway) it's less taxing on the cpu because you are not doing that second geometry pass. Again this may not lead to any FPS increase at all but using less cpu can never be a bad thing. I guess when the project is a little more advanced some detailed profiling and benchmarking will be in order.
#6
06/22/2014 (5:25 pm)
I'm will to try it out :)!! 
#7
06/22/2014 (7:20 pm)
That sounds like very good work, and a welcome upgrade. Consider me interested. 
#8
06/22/2014 (9:16 pm)
As the husband of a history major, I can confidently say knowing the origin greatly improves the understanding of today and tomorrow. Torque 3D's lighting system was based on Wolfgang Engel's white papers. He even acknowledge it in person. If you study his presentations and how T3D implements it, you will have a more high level understanding of how it all works. From there, you can deconstruct it, improve it, and maybe even break it for future improvements. 
#9
Not true.
Some meshes are rendered 3 or more times, depending on buffer information.
Glow objects are pushed 3 times in different buffers (bins).
"..were using deferred lighting have made the change to deferred shading ( see: UE4 ).
Not true.
Unreal does not use deferred rendering at all. The use forward pass for details. Forward rendering leads to better quality, due to how rasterizers work. They use FBM and project the frequency to produce colors.
Transparent objects still use a forward pass.
T3D does not render everything to different buffers.
You can see in the code "condition" and "uncondition". It means pack/unpack. Information is packed in order to reduce rtts , fillrate and memory at all.
Finally T3D is producing dependent screen space sampling to collect the buffer information and render the final passes.
06/23/2014 (2:06 am)
"Everything is rendered twice"Not true.
Some meshes are rendered 3 or more times, depending on buffer information.
Glow objects are pushed 3 times in different buffers (bins).
"..were using deferred lighting have made the change to deferred shading ( see: UE4 ).
Not true.
Unreal does not use deferred rendering at all. The use forward pass for details. Forward rendering leads to better quality, due to how rasterizers work. They use FBM and project the frequency to produce colors.
Transparent objects still use a forward pass.
T3D does not render everything to different buffers.
You can see in the code "condition" and "uncondition". It means pack/unpack. Information is packed in order to reduce rtts , fillrate and memory at all.
Finally T3D is producing dependent screen space sampling to collect the buffer information and render the final passes.
#10
From Official UE4 documentation and one epic games employee:
Source: Official UE4 documentation
Link: docs.unrealengine.com/latest/INT/Engine/Rendering/Overview/index.html
Source: Brian Karis @ Epic Games
Link: blog.selfshadow.com/publications/s2013-shading-course/karis/s2013_pbs_epic_notes...
06/23/2014 (3:27 am)
Quote:Not true.
Unreal does not use deferred rendering at all.
From Official UE4 documentation and one epic games employee:
Quote:
All lights are applied deferred in Unreal Engine 4, as opposed to the forward lighting path used in Unreal Engine 3. Materials write out their attributes into the GBuffers, and lighting passes read in the per-pixel material properties and perform lighting with them.
Source: Official UE4 documentation
Link: docs.unrealengine.com/latest/INT/Engine/Rendering/Overview/index.html
Quote:
Deferred shading limits the number of shading models we can have, so our base shading model
needs to be descriptive enough to cover 99% of the materials that occur in the real world.
Quote:
For the future we are considering adopting a hybrid deferred/forward shading approach to better support
more specialized shading models. Currently with our purely deferred shading approach, different
shading models are handled with a dynamic branch from a shading model id stored in the G-Buffer
Source: Brian Karis @ Epic Games
Link: blog.selfshadow.com/publications/s2013-shading-course/karis/s2013_pbs_epic_notes...
#11
06/23/2014 (6:56 am)
Great job, you can count me in.
#12
YES! Please add this to the main repo whenever you consider it stable enough to do so.
06/23/2014 (10:27 am)
Thanks for all of the hard work on this guys. If given a vote, I would vote:YES! Please add this to the main repo whenever you consider it stable enough to do so.
#13
@Ivan
Those are additional rendering features that will be present in either system. When you shut off all the extra stuff like glow and transparency, generally speaking, every object is rendered twice. This may be the first time some users have ever heard of deferred shading or lighting so I was trying to keep it as simple as possible.
@Ivan
That's an overly technical, though accurate, explanation of how things currently work in Torque, but I'm not sure why you're posting it. We've already dug into Torque's rendering and lighting and figured all this out. Then we turned it into deferred shading :D
06/23/2014 (2:05 pm)
Quote:
Not true.
Some meshes are rendered 3 or more times, depending on buffer information.
Glow objects are pushed 3 times in different buffers (bins)."
@Ivan
Those are additional rendering features that will be present in either system. When you shut off all the extra stuff like glow and transparency, generally speaking, every object is rendered twice. This may be the first time some users have ever heard of deferred shading or lighting so I was trying to keep it as simple as possible.
Quote:
T3D does not render everything to different buffers.
You can see in the code "condition" and "uncondition". It means pack/unpack. Information is packed in order to reduce rtts , fillrate and memory at all.
Finally T3D is producing dependent screen space sampling to collect the buffer information and render the final passes.
@Ivan
That's an overly technical, though accurate, explanation of how things currently work in Torque, but I'm not sure why you're posting it. We've already dug into Torque's rendering and lighting and figured all this out. Then we turned it into deferred shading :D
#14
06/23/2014 (3:45 pm)
I got everything compiled error free but it crashes every time I start or try to start a mission...Any idea of what the issue could be?
#15
Are you running the testDS project in the My Projects folder or did you generate a new one? I should have mentioned in the original post that we haven't migrated changes from testDS project into the templates yet, so they're missing shaders and will likely crash. We keep making changes to the shaders so once they're more finalized we'll put together a release for testing. In the mean time the latest and greatest is in My Projects\testDS.
06/23/2014 (3:49 pm)
@ KoryAre you running the testDS project in the My Projects folder or did you generate a new one? I should have mentioned in the original post that we haven't migrated changes from testDS project into the templates yet, so they're missing shaders and will likely crash. We keep making changes to the shaders so once they're more finalized we'll put together a release for testing. In the mean time the latest and greatest is in My Projects\testDS.
#16
I got a chance to test it..Everything seem pretty standard and worked for the most part..Of course no sky in scattersky but I'm sure through time that will be fixed.
Everything runs about the same maybe a slight bit faster but my computer is crap. I can't really do any hardcore testing until I get another one. Great work guys..Now I'm amped to see what you guys come up with in the way of PBR :)
06/24/2014 (6:06 am)
AndrewI got a chance to test it..Everything seem pretty standard and worked for the most part..Of course no sky in scattersky but I'm sure through time that will be fixed.
Everything runs about the same maybe a slight bit faster but my computer is crap. I can't really do any hardcore testing until I get another one. Great work guys..Now I'm amped to see what you guys come up with in the way of PBR :)
#17
They still use mixed shading. They cannot remove forward shading, because the whole mobile industry rely on forward shading :)
@Andrew
My point is that you do not need to render objects twice in order to achieve PBR. In the common case for deferred rendering, all shadow renderers for example are dependent renderers.
If you need albedo, specular, BRDF params or something for your shaders, then you draw and sample the buffer.
The biggest problem is not forward vs deferred shading, but the light composition and decomposition. I lost a lot of time on color blending in order to achieve realistic results.
You start a project, that finished for me in 2012. That's why I am writing this.
06/24/2014 (7:04 am)
@TimmyThey still use mixed shading. They cannot remove forward shading, because the whole mobile industry rely on forward shading :)
@Andrew
My point is that you do not need to render objects twice in order to achieve PBR. In the common case for deferred rendering, all shadow renderers for example are dependent renderers.
If you need albedo, specular, BRDF params or something for your shaders, then you draw and sample the buffer.
The biggest problem is not forward vs deferred shading, but the light composition and decomposition. I lost a lot of time on color blending in order to achieve realistic results.
You start a project, that finished for me in 2012. That's why I am writing this.
#18
Present design thinking is for the additional data requirements, sort those into a secondary buffer concurrently with lighting, and recombine afterwards via postfx, leaving things like translucency passes and other render managers largely untouched for now, rather than the present system of rendering one pass for lighting data and going back over it with a diffuse-modified redraw. Clearer?
06/24/2014 (10:11 am)
@Ivan: (Still trying to keep this high level enough for those unfamiliar with the systems) Present design thinking is for the additional data requirements, sort those into a secondary buffer concurrently with lighting, and recombine afterwards via postfx, leaving things like translucency passes and other render managers largely untouched for now, rather than the present system of rendering one pass for lighting data and going back over it with a diffuse-modified redraw. Clearer?
#19
I don't know what you're advocating. It almost sounds like you want to switch torque to forward rendering and nothing else. That's just not what we're hear to do. This thread is about discussing the pros, cons, and/or development of deferred shading. We left the forward rendering options in tact and currently utilize them for anything that can't be done through deferred.
Regarding PBR, you're right it's not absolutely necessary to switch to deferred shading to achieve it, but I haven't heard a solid argument yet to stick to deferred lighting. Two separate teams of highly skilled engineers at CryTek and Epic decided it was best to switch to deferred rendering to achieve the best PBR results, and we're going to follow suit.
06/24/2014 (3:22 pm)
@ IvanI don't know what you're advocating. It almost sounds like you want to switch torque to forward rendering and nothing else. That's just not what we're hear to do. This thread is about discussing the pros, cons, and/or development of deferred shading. We left the forward rendering options in tact and currently utilize them for anything that can't be done through deferred.
Regarding PBR, you're right it's not absolutely necessary to switch to deferred shading to achieve it, but I haven't heard a solid argument yet to stick to deferred lighting. Two separate teams of highly skilled engineers at CryTek and Epic decided it was best to switch to deferred rendering to achieve the best PBR results, and we're going to follow suit.
#20
http://gameangst.com/?p=141
Lots of reasons to go with a fully deferred rendering solution.
06/24/2014 (7:43 pm)
This is a very interesting article from 2009.http://gameangst.com/?p=141
Lots of reasons to go with a fully deferred rendering solution.
raa brubb
So the only thing I was wondering is, does it differ on the renderer? So, will you have to change things depending if it is OpenGL or Direct3D?