How best to extend Rendering pipeline with custom code/shader technique
by Edward Rotberg · in Torque 3D Professional · 12/07/2009 (1:34 pm) · 36 replies
Hi,
I'm totally new to Torque, 3D or otherwise, though I have extensive experience in the video games business and have written many rendering engines myself and used a number of others. For the project I am on now, we have decided to use Torque3D, and I must say that I'm very impressed with what I have seen so far. I have started looking through the source code of the engine, and everything seems well laid out. However, it is a lot to get your head around and time is not on my side here. As a result I'm hoping to get some pointers to fast track me towards my current task.
This task is to integrate into Torque3D's rendering pipeline, a custom rendering technique. To make things a little simpler, this will only need to work for the PC platform, and thus can be completely Direct3D9-centric. The requirements of this technique include custom setup and management of vertex buffers and texture buffers as well as streamed instancing. At present there is no need to do collision with the instances to be rendered, so that is something that can be disregarded. Obviously, the actual rendering code for this will need to be largely written in C++ with Direct3D API calls and will use custom shader code (currently written in HLSL).
From what I can tell, in going though the documentation and what little of the source I've been able to examine so far (I've only had the source code for one day), it looks like, at the very least, I will need to write a RenderBinManager and/or a RenderDelegate. However, because of the platform specific nature of this code, I'm not certain that this is where I will need to hook in.
For the RenderBinManager, I'm also not certain whether it would be advantageous to subclass RenderMeshMgr, or write the manager from scratch. I will have to load skinned mesh assets as well as textures and other custom binary data, and create custom format vector (FVF) and texture buffers. I will also probably need to manage instance culling unless there is some fast, outboard culling capability somewhere that I can leverage. There is also the matter of hooking into the Update loop (as opposed to the Rendering loop) for other frame dependent updates besides culling.
So my primary question is this: Is there some sample code somewhere that demonstrates how to do something along these lines? If not, can someone provide pointers or other enlightenment for this task?
Any help you can afford is greatly appreciated. Thanks in advance,
= Ed Rotberg =
I'm totally new to Torque, 3D or otherwise, though I have extensive experience in the video games business and have written many rendering engines myself and used a number of others. For the project I am on now, we have decided to use Torque3D, and I must say that I'm very impressed with what I have seen so far. I have started looking through the source code of the engine, and everything seems well laid out. However, it is a lot to get your head around and time is not on my side here. As a result I'm hoping to get some pointers to fast track me towards my current task.
This task is to integrate into Torque3D's rendering pipeline, a custom rendering technique. To make things a little simpler, this will only need to work for the PC platform, and thus can be completely Direct3D9-centric. The requirements of this technique include custom setup and management of vertex buffers and texture buffers as well as streamed instancing. At present there is no need to do collision with the instances to be rendered, so that is something that can be disregarded. Obviously, the actual rendering code for this will need to be largely written in C++ with Direct3D API calls and will use custom shader code (currently written in HLSL).
From what I can tell, in going though the documentation and what little of the source I've been able to examine so far (I've only had the source code for one day), it looks like, at the very least, I will need to write a RenderBinManager and/or a RenderDelegate. However, because of the platform specific nature of this code, I'm not certain that this is where I will need to hook in.
For the RenderBinManager, I'm also not certain whether it would be advantageous to subclass RenderMeshMgr, or write the manager from scratch. I will have to load skinned mesh assets as well as textures and other custom binary data, and create custom format vector (FVF) and texture buffers. I will also probably need to manage instance culling unless there is some fast, outboard culling capability somewhere that I can leverage. There is also the matter of hooking into the Update loop (as opposed to the Rendering loop) for other frame dependent updates besides culling.
So my primary question is this: Is there some sample code somewhere that demonstrates how to do something along these lines? If not, can someone provide pointers or other enlightenment for this task?
Any help you can afford is greatly appreciated. Thanks in advance,
= Ed Rotberg =
About the author
#2
I do appreciate the link though as I'll have to address the shader side of things as well as the rest of the code.
= Ed =
12/07/2009 (3:37 pm)
Thanks for the link Eric. It is very informative and there is a lot of stuff there that will be good to know as I go forward with this task. Unfortunately, it does not at all address the actual instanced rendering that will need to be done, since this is not simply a matter of creating a custom shader to provide a new material. Additionally, I will need to know how to provide secondary and possibly tertiary vertex buffers to the vertex shader portion of the custom shader code. But all of that will still rely on custom rendering code to feed other appropriate data on a frame by frame basis for a custom rendering stage. It cannot simply be done via a shader.I do appreciate the link though as I'll have to address the shader side of things as well as the rest of the code.
= Ed =
#3
I guess in most of everything you've described Torque 3D does, but the features and hooks are spread about the systems.
First off if it was me i would try to work with the exiting abstracted classes as much as possible and poke in new overloaded functionality where i need it. This will make it easier for you to merge future updates and take advantage of new features/fixes in 1.1 and 1.2.
Let me now try to hit all your issues really quick to give you a starting point. Note i'm in the middle of installing a new OS, so i'm doing this from memory and don't have code infront of me.
As far as instancing, assuming you mean dx9 mesh instancing, we don't currently have support for it or multiple vertex streams in the GFX API, but we're working on it now... maybe 1.1 beta if things go well.
Ideally the data your trying to render fits into one of the existing bins and it can draw it for you... but it sounds like thats not the case here.
If its all seperate files then make a few different resource classes where needed.... if your loading textures you can of course reuse what we got already.
As far as FVFs... the GFXVertexFormat class is used to define new formats. See the header file for macros and examples of how to define a new format for use with GFXVertexBuffers.
(...)
12/08/2009 (3:17 am)
Hi Ed.Quote:I'm totally new to Torque, 3D or otherwise, though I have extensive experience in the video games business and have written many rendering engines myself and used a number of others.If you are who your name suggests then... yea... you might know a thing or two. ;)
Quote:As a result I'm hoping to get some pointers to fast track me towards my current task.Should be no problem.
I guess in most of everything you've described Torque 3D does, but the features and hooks are spread about the systems.
First off if it was me i would try to work with the exiting abstracted classes as much as possible and poke in new overloaded functionality where i need it. This will make it easier for you to merge future updates and take advantage of new features/fixes in 1.1 and 1.2.
Let me now try to hit all your issues really quick to give you a starting point. Note i'm in the middle of installing a new OS, so i'm doing this from memory and don't have code infront of me.
Quote:The requirements of this technique include custom setup and management of vertex buffers and texture buffers as well as streamed instancing.Torque already has some fairly low level access to vertex buffers and textures.... see GFXVertexBufferHandle, GFXTexHandle, GFXTextureObject, and GFXTextureManager. If you look at the TSForestCellBatch and maybe TSLastDetail classes you can see examples of manipulation of vertex and texture buffers.
As far as instancing, assuming you mean dx9 mesh instancing, we don't currently have support for it or multiple vertex streams in the GFX API, but we're working on it now... maybe 1.1 beta if things go well.
Quote:I will need to write a RenderBinManager and/or a RenderDelegate. However, because of the platform specific nature of this code, I'm not certain that this is where I will need to hook in.Depends on what your doing. A RenderDelegate would be good enough to get you one callback hook when your ready to do your rendering. A new bin would be good if your rendering is complex and probably needs to be seperate from your generation/gamplay logic anyway.
Ideally the data your trying to render fits into one of the existing bins and it can draw it for you... but it sounds like thats not the case here.
Quote:For the RenderBinManager, I'm also not certain whether it would be advantageous to subclass RenderMeshMgr, or write the manager from scratch.You can do either and it should work fine. If your needs are really specialized you might as well start from scratch and see where it leads you.
Quote: I will have to load skinned mesh assets as well as textures and other custom binary data, and create custom format vector (FVF) and texture buffers.Sounds like this is all some custom model format. If its all coming out of one file i would look at making a class for loading and assembling this data and make the object a resource instantiated by the resource manager.
If its all seperate files then make a few different resource classes where needed.... if your loading textures you can of course reuse what we got already.
As far as FVFs... the GFXVertexFormat class is used to define new formats. See the header file for macros and examples of how to define a new format for use with GFXVertexBuffers.
(...)
#4
Your top level SceneObject derived class will get culling itself if it has a non-global bounding box ( see SceneObject::setGlobalBounds() ). After that the SceneState class passed to the prepRenderImage function will contain a Frustum object, SceneState::getFrustum(), which has various culling methods.
On background the prepRenderImage method (which is named after some crazy legacy stuff) is called so you can submit RenderInsts to the bins. You shouldn't render from there directly, but prep your bins to do so. So culling usually occurs here.
If your composite class derives from GameBase you'll get callbacks for ticks, interpolate ticks, and other stuff like datablock support... a datablock being like a holder class for reducing network bandwidth and sharing resources.
Another option is to also inherit from the IProcessTick class which will get the ticks and interpolation calls without any other gameplay stuff.
What are your needs for shaders? Do you need to fit into the Advanced Lighting system for deferred lights and shadows?
12/08/2009 (3:18 am)
(Continued...)Quote:I will also probably need to manage instance culling unless there is some fast, outboard culling capability somewhere that I can leverage.It sounds like your trying to create a composite object to render a bunch of other objects. For sure look at the Forest code as it does the same sort of thing to manage 100s of thousands of trees.
Your top level SceneObject derived class will get culling itself if it has a non-global bounding box ( see SceneObject::setGlobalBounds() ). After that the SceneState class passed to the prepRenderImage function will contain a Frustum object, SceneState::getFrustum(), which has various culling methods.
On background the prepRenderImage method (which is named after some crazy legacy stuff) is called so you can submit RenderInsts to the bins. You shouldn't render from there directly, but prep your bins to do so. So culling usually occurs here.
Quote:There is also the matter of hooking into the Update loop (as opposed to the Rendering loop) for other frame dependent updates besides culling.Two options.
If your composite class derives from GameBase you'll get callbacks for ticks, interpolate ticks, and other stuff like datablock support... a datablock being like a holder class for reducing network bandwidth and sharing resources.
Another option is to also inherit from the IProcessTick class which will get the ticks and interpolation calls without any other gameplay stuff.
Quote:Is there some sample code somewhere that demonstrates how to do something along these lines? If not, can someone provide pointers or other enlightenment for this task?From all you've described i would look at the Forest classes and maybe MeshRoad and River. They should give you a good idea of how to do culling and rendering tasks.
What are your needs for shaders? Do you need to fit into the Advanced Lighting system for deferred lights and shadows?
#5
Thanks so much for the detailed reply. You've given me a lot to look at, and I'll be digging into it all today - By the way - where can I find the Forest code you referred to? I did not see it in the Engine Source, though I saw about 3 or 4 references to "forest"?
Let me answer a couple of the questions you raised.
As for my name, I am he (and he is me). Your assumption that I'm trying to create a composite object to render a bunch of others is a fair way to think of this. In our stand-alone tests we had to do some arm twisting of DX9 to get all this stuff to work, so we have a working, DirectX version of the code we are trying to integrate.
You asked about shaders, and of course, given the multiple vertex buffers and such, this will all involve some custom shaders, which we have currently written in HLSL. At present, we are most interested in just getting our code up and running under Torque3D, but eventually it would be great to fit it into the Advanced Lighting System at least for deferred lights (we may not need shadows), assuming that this would not significantly impact the performance of our technique.
At present, I'm wondering if an incremental approach to getting this integrated into Torque3D might not be to first just put everything into a custom RenderDelegate, including all of the Direct3D9 calls, vertex and texture buffer management, and resource loading in our own formats, as well as the Update processing just to get it all rendering under Torque3D as quickly as possible (an important milestone for us). Then I can start to migrate various sections of the code to their "proper" places under Torque3D - for example moving the asset loading to the ResourceManager, managing our Vertex and Texture Buffers with GFXVertexBufferHandles and GFXTextureObjects via the Texture Manager, moving the Update code to process under Tick management, etc.
Does this approach make sense to you or do you think it will cause more problems than it solves?
Thanks again for the speedy response and the great pointers.
= Ed =
12/08/2009 (12:05 pm)
Tom,Thanks so much for the detailed reply. You've given me a lot to look at, and I'll be digging into it all today - By the way - where can I find the Forest code you referred to? I did not see it in the Engine Source, though I saw about 3 or 4 references to "forest"?
Let me answer a couple of the questions you raised.
As for my name, I am he (and he is me). Your assumption that I'm trying to create a composite object to render a bunch of others is a fair way to think of this. In our stand-alone tests we had to do some arm twisting of DX9 to get all this stuff to work, so we have a working, DirectX version of the code we are trying to integrate.
You asked about shaders, and of course, given the multiple vertex buffers and such, this will all involve some custom shaders, which we have currently written in HLSL. At present, we are most interested in just getting our code up and running under Torque3D, but eventually it would be great to fit it into the Advanced Lighting System at least for deferred lights (we may not need shadows), assuming that this would not significantly impact the performance of our technique.
At present, I'm wondering if an incremental approach to getting this integrated into Torque3D might not be to first just put everything into a custom RenderDelegate, including all of the Direct3D9 calls, vertex and texture buffer management, and resource loading in our own formats, as well as the Update processing just to get it all rendering under Torque3D as quickly as possible (an important milestone for us). Then I can start to migrate various sections of the code to their "proper" places under Torque3D - for example moving the asset loading to the ResourceManager, managing our Vertex and Texture Buffers with GFXVertexBufferHandles and GFXTextureObjects via the Texture Manager, moving the Update code to process under Tick management, etc.
Does this approach make sense to you or do you think it will cause more problems than it solves?
Thanks again for the speedy response and the great pointers.
= Ed =
#6
Custom shaders via CustomMaterial in T3D are tricky at the moment to get working with AL, but if thats not a need then it should work ok for you.
12/08/2009 (7:37 pm)
Quote:where can I find the Forest code you referred to?I suspect your using T3D 1.0.1... the forest code is actually in the 1.1 alpha along with a bunch of other new stuff. You should at least grab it and examine the code as i believe most of it all still applies with 1.0.1.
Quote:Does this approach make sense to you or do you think it will cause more problems than it solves?If you have it already working in a stand alone DX9 app then yea... i would render it yourself from a render delegate callback. You should be able to cast some classes to get into the DX9 specific versions and get at the internals needed to get things working.
Custom shaders via CustomMaterial in T3D are tricky at the moment to get working with AL, but if thats not a need then it should work ok for you.
#7
That's pretty much what I figured. I've downloaded the Alpha and will have a look at the Forest code when I get over the hump with making this work as a hack. In the meantime, maybe you can answer another question for me. Where should I located the code I'm adding? I tried putting thesouce files that I added into the "My Project\testProj\source" folder, and adding them to the Visual Studio solution for the project under the testProj project under Source Files->source. Unfortunately, even though I have fixed all of the compile errors, I am through tons of unresolved external errors from the link, including what looks like all of the DX9 calls, as well as Torque3D classes that I'm using.
I have a feeling that I've added this stuff to the wrong location, but I'm just not sure where it all should go.
Thanks in advance,
= Ed =
12/08/2009 (9:04 pm)
Thanks Tom,That's pretty much what I figured. I've downloaded the Alpha and will have a look at the Forest code when I get over the hump with making this work as a hack. In the meantime, maybe you can answer another question for me. Where should I located the code I'm adding? I tried putting thesouce files that I added into the "My Project\testProj\source" folder, and adding them to the Visual Studio solution for the project under the testProj project under Source Files->source. Unfortunately, even though I have fixed all of the compile errors, I am through tons of unresolved external errors from the link, including what looks like all of the DX9 calls, as well as Torque3D classes that I'm using.
I have a feeling that I've added this stuff to the wrong location, but I'm just not sure where it all should go.
Thanks in advance,
= Ed =
#8
I suspect you put them in the source folder, but you added them to the your 'testProj' project and not the 'testProj DLL' project. Its sort of weird, but it was required for browser plugin stuff and other embedding reasons to be that way.
If your unsure just put them in the source folder and then run the generateProjects.bat file and it will rebuild the solution/projects and make sure your source is included in the right project.
As far as missing DirectX headers... the build system assumes that the DX include and libs paths are set in Visual Studio already under Tools->Options->Projects and Solutions->VC++ Directories. You just need to add the location of the includes and libs/x86 folders.
12/08/2009 (11:29 pm)
Did you add them to the VisualStudio project manually?I suspect you put them in the source folder, but you added them to the your 'testProj' project and not the 'testProj DLL' project. Its sort of weird, but it was required for browser plugin stuff and other embedding reasons to be that way.
If your unsure just put them in the source folder and then run the generateProjects.bat file and it will rebuild the solution/projects and make sure your source is included in the right project.
As far as missing DirectX headers... the build system assumes that the DX include and libs paths are set in Visual Studio already under Tools->Options->Projects and Solutions->VC++ Directories. You just need to add the location of the includes and libs/x86 folders.
#9
You're a savior once again. I did what you suggested and all of the link errors went away.
Now, unfortunately, I have no idea how to add this new RenderObject to the game. I edited the renderManager.cs script and added the following line to the appropriate place.
DiffuseRenderPassManager.addManager( new RenderObjectMgr() { bintype = "Custom"; renderOrder = 0.95; processAddOrder = 0.95; } );
But that didn't seem to help. I put breakpoints in all of the functions that I wrote for this object, but only initPersistFields() ever gets called. I tried to find the object in the world editor to add it to the level, but I could not find it to add. I've modeled my RenderObject after both renderObjectExample.cpp/.h and skyBox.cpp/.h. The only functions I have implemented in my RenderObject "Custom" are:
_initRender - called internally from onAdd to build my vertex & texture buffers as well as loading my custom shader and other things
_renderObject - registered as my RenderDelegate from prepRenderImage
Custom - class constructor initializes mNetFlags & mTypeFlags
~Custom - class destructor. Currently does nothing but eventually I'll have to release some buffers here.
initPersistFields - just calls Parent.
onAdd - looks like renderObjectExample's version but with a much larger bounding box. It also calls _initRender
onRemove - just like the call in renderObjectExample
prepRenderImage - this is based more on the skyBox example, for no apparent reason. renderObjectExample initializes its geometry here where I do it in onAdd.
This is all probably very trivial stuff, but I'm feeling a bit lost at this point. Thanks in advance for clearing the fog.
= Ed =
12/09/2009 (1:18 pm)
Tom,You're a savior once again. I did what you suggested and all of the link errors went away.
Now, unfortunately, I have no idea how to add this new RenderObject to the game. I edited the renderManager.cs script and added the following line to the appropriate place.
DiffuseRenderPassManager.addManager( new RenderObjectMgr() { bintype = "Custom"; renderOrder = 0.95; processAddOrder = 0.95; } );
But that didn't seem to help. I put breakpoints in all of the functions that I wrote for this object, but only initPersistFields() ever gets called. I tried to find the object in the world editor to add it to the level, but I could not find it to add. I've modeled my RenderObject after both renderObjectExample.cpp/.h and skyBox.cpp/.h. The only functions I have implemented in my RenderObject "Custom" are:
_initRender - called internally from onAdd to build my vertex & texture buffers as well as loading my custom shader and other things
_renderObject - registered as my RenderDelegate from prepRenderImage
Custom - class constructor initializes mNetFlags & mTypeFlags
~Custom - class destructor. Currently does nothing but eventually I'll have to release some buffers here.
initPersistFields - just calls Parent.
onAdd - looks like renderObjectExample's version but with a much larger bounding box. It also calls _initRender
onRemove - just like the call in renderObjectExample
prepRenderImage - this is based more on the skyBox example, for no apparent reason. renderObjectExample initializes its geometry here where I do it in onAdd.
This is all probably very trivial stuff, but I'm feeling a bit lost at this point. Thanks in advance for clearing the fog.
= Ed =
#10
My head hurts.
= Ed =
12/09/2009 (1:48 pm)
For that matter, I don't understand how it is calling that member function initPersistFields BEFORE it calls the constructor!My head hurts.
= Ed =
#11
RE initPersistFields:
This is per-class functionality, i.e. the field layout of a SimObject class is registered once with the console system at startup.
12/09/2009 (1:52 pm)
RE initPersistFields:
This is per-class functionality, i.e. the field layout of a SimObject class is registered once with the console system at startup.
#12
That makes some sense. At this point, I think the problem is that I haven't been able to add my object to the world. I'm sure once I figure that out I'll have lots of problems with my own code, but I'm still a little bit off from that at the moment.
I can see both the skyBox object (under Library->Level->Environment) and the renderObjectExample (under Library->Level->Examples). But I cannot see my Custom object anywhere. I have tried setting TypeMask |= EnvironmentObjectType and mTypeMask |= StaticObjectType, but neither of those work.
Still hacking away at it. If anyone has a suggestion...
Thanks again,
= Ed =
12/09/2009 (2:17 pm)
Thanks Rene,That makes some sense. At this point, I think the problem is that I haven't been able to add my object to the world. I'm sure once I figure that out I'll have lots of problems with my own code, but I'm still a little bit off from that at the moment.
I can see both the skyBox object (under Library->Level->Environment) and the renderObjectExample (under Library->Level->Examples). But I cannot see my Custom object anywhere. I have tried setting TypeMask |= EnvironmentObjectType and mTypeMask |= StaticObjectType, but neither of those work.
Still hacking away at it. If anyone has a suggestion...
Thanks again,
= Ed =
#13
These particular tabs are not created procedurally but rather manually in the editor code. To add objects to them, look at tools/worldEditor/scripts/editors/creator.ed.cs. The first function in the file does all the setup there.
12/09/2009 (2:24 pm)
These particular tabs are not created procedurally but rather manually in the editor code. To add objects to them, look at tools/worldEditor/scripts/editors/creator.ed.cs. The first function in the file does all the setup there.
#14
Thanks again.
= Ed =
12/09/2009 (2:40 pm)
Thanks Rene. I've added my object there, and lo and behold it shows up in the world editor! However, when I try to add an instance of this object, the World Editor crashes with no useful information. Looks like I'll be applying for some Torque paid support soon.Thanks again.
= Ed =
#15
12/09/2009 (3:45 pm)
Just to interject something possibly helpful... to catch useful info before Torque crashes, simply put a break point on line 17 of "Engine/source/platformWin32/winProcessControl.cpp" which reads "DebugBreak();". This will break while there is still useful info in the stack, which you can traverse to track down the culprit.
#16
I would suspect that the custom object type you have added is directly or indirectly triggering a crash in its init code (constructor or onAdd). Have you run this in the debugger?
12/09/2009 (3:45 pm)
I would suspect that the custom object type you have added is directly or indirectly triggering a crash in its init code (constructor or onAdd). Have you run this in the debugger?
#17
Thanks again,
= Ed =
12/09/2009 (3:59 pm)
Thanks for both of these responses. I will definitely put a breakpoint where you suggest Ryan. As for running this in the debugger, can you tell me how I can run the World Editor in the debugger? I'm certain the problem is with my code, but I can't figure out how to debug it via the World Editor. I can run the application from the debugger, but how do I instantiate my object other than from the World Editor?Thanks again,
= Ed =
#18
Simply compile a debug executable and run that from the IDE. Then drop into the world editor with F11 and take it to the crash.
To instantiate your object other than from the World Editor UI, use the scripting abilities of Torque.
Pull down the console with ` and type "new MyCustomObjectType();". Be aware that if it's a ShapeBase-derived class, it needs a datablock or it'll fail its onAdd.
12/09/2009 (4:05 pm)
Simply compile a debug executable and run that from the IDE. Then drop into the world editor with F11 and take it to the crash.
To instantiate your object other than from the World Editor UI, use the scripting abilities of Torque.
Pull down the console with ` and type "new MyCustomObjectType();". Be aware that if it's a ShapeBase-derived class, it needs a datablock or it'll fail its onAdd.
#19
Aha. Thanks! It is a RenderObject based class, so the datablock is not a problem. I entered the World editor as you suggested, and now I'm on my way! I was unaware of the F11 trick to get to the Wold Editor.
Thanks for the tip!
= Ed =
12/09/2009 (4:14 pm)
Rene,Aha. Thanks! It is a RenderObject based class, so the datablock is not a problem. I entered the World editor as you suggested, and now I'm on my way! I was unaware of the F11 trick to get to the Wold Editor.
Thanks for the tip!
= Ed =
#20
If you hit an error in T3D and you don't have a valid stack, try hitting Shift+F11 (Visual Studio's default key for 'Step Out') two or three times, and you should step back to a valid stack.
There is a better way to do a DebugBreak() that doesn't screw up the stack like this, but I can't remember it for the life of me.
12/09/2009 (4:33 pm)
@Ryan (and others):If you hit an error in T3D and you don't have a valid stack, try hitting Shift+F11 (Visual Studio's default key for 'Step Out') two or three times, and you should step back to a valid stack.
There is a better way to do a DebugBreak() that doesn't screw up the stack like this, but I can't remember it for the life of me.
Torque 3D Owner Eric Armstrong
Bloody Tongue Games
gameclay.com/documentation/t3d-materials