Sahara for Torque 3D MIT 3.5.1...
by Quinton Delpeche · in Torque 3D Professional · 07/11/2014 (5:24 am) · 43 replies
I am pleased to announce that Konrad Kiss has decided to allow me to release this to the Torque 3D MIT community via a pull request. He has stated that any licensing questions should be directed at him via @KonradKiss.
I am busy doing the final integration testing now and I have a last few steps to check before I am ready to do the pull request. But before I get ahead of myself, let me just give you all some further information.
You can get more information from the BitGap Games website: www.bitgap.com/products/12/sahara.html
So where are we right now? Well I have all the engine changes merged in and I have a test build running, I am currently doing my last tests to make sure that it is stable ... luckily Konrad's code is top notch and there haven't been any gotchas.
What can you do? Well my Git knowledge is a little fuzzy ... almost non-existent ... so I need some help on doing the Pull request for this code. So if anyone can assist me here, I will get the Pull request done and then do a new blog with some updates.
There are also two extra source files that need to be added "accuFeature.cpp" and "accuFeature.h", currently these reside in a folder called "sahara" in the root of the "Engine/source/" folder. Where ideally should these files be placed and/or renamed to better fit in with the current coding standards.
So if anyone can help me getting this Pull request done ... and feel free to send Konrad some thank yous ... this is very generous of him.
Edit:
I have created a GitHub repository for this and I am waiting for the Steering Committee to approve the Pull Request. The repository can be found here.
I have also created a resource for the time being, read it here.
I am busy doing the final integration testing now and I have a last few steps to check before I am ready to do the pull request. But before I get ahead of myself, let me just give you all some further information.
Quote:Sahara is a tech extension to the Torque 3D 1.1 game engine ( 1.1 Beta 2 and later versions) by GarageGames which lets you create more realistic game environments by generating naturally accumulated dirt, snow, dust or any other material over your game shapes. It is a successor to our Cliff Construction Kit.
The motto of Sahara is: use one model in many environments. Sahara can be set up on a per material basis. Once installed, use the Torque 3D Material Editor to edit the properties of your shape.
You can get more information from the BitGap Games website: www.bitgap.com/products/12/sahara.html
So where are we right now? Well I have all the engine changes merged in and I have a test build running, I am currently doing my last tests to make sure that it is stable ... luckily Konrad's code is top notch and there haven't been any gotchas.
What can you do? Well my Git knowledge is a little fuzzy ... almost non-existent ... so I need some help on doing the Pull request for this code. So if anyone can assist me here, I will get the Pull request done and then do a new blog with some updates.
There are also two extra source files that need to be added "accuFeature.cpp" and "accuFeature.h", currently these reside in a folder called "sahara" in the root of the "Engine/source/" folder. Where ideally should these files be placed and/or renamed to better fit in with the current coding standards.
So if anyone can help me getting this Pull request done ... and feel free to send Konrad some thank yous ... this is very generous of him.
Edit:
I have created a GitHub repository for this and I am waiting for the Steering Committee to approve the Pull Request. The repository can be found here.
I have also created a resource for the time being, read it here.
About the author
Gobbo Games is an Independent Games Development company operating from Durban in South Africa. We believe in creating high-quality cost-effective games that remain true to the belief of Independent Game Developers around the world.
#2
I meant to MIT Sahara and hoped to have a chance to integrate it into the engine for a good while now, but things have been way too busy and I never got around to getting started. If and when it's accepted, I hope that people working with the engine will find Sahara useful.
I'll update the website as well once it's pulled into the development branch. Existing customers of Sahara can choose to keep their licenses (for the T3D versions that we originally supported, although MIT is a more permissive license). Our support for all Bitgap products will likely end sometime in the coming months. The bitgap.com site will probably be retired soon as well. I'll keep everyone updated, so that you have a chance to download all your products before that happens. :)
Thanks again! :)
07/11/2014 (5:55 am)
Thanks for doing this, Quinton, I immensely appreciate your help!I meant to MIT Sahara and hoped to have a chance to integrate it into the engine for a good while now, but things have been way too busy and I never got around to getting started. If and when it's accepted, I hope that people working with the engine will find Sahara useful.
I'll update the website as well once it's pulled into the development branch. Existing customers of Sahara can choose to keep their licenses (for the T3D versions that we originally supported, although MIT is a more permissive license). Our support for all Bitgap products will likely end sometime in the coming months. The bitgap.com site will probably be retired soon as well. I'll keep everyone updated, so that you have a chance to download all your products before that happens. :)
Thanks again! :)
#3
07/11/2014 (6:10 am)
That's awfully kind of you, Mr Kiss. :)
#4
07/11/2014 (6:10 am)
@Jesse: I believe that the hit is negligible (should depend on the size and count of your textures, but still). The core feature is a slim shader, so I believe you should only see it lose a few frames on older cards. Although truth be told, it was never properly measured - I never saw the need. I haven't received any complaints about performance either, so that should be a good sign. :)
#5
07/11/2014 (6:11 am)
Mr. Acaster! The pleasure is all mine. :)
#6
I have used the original commercial version before and never really experienced any performance hits.
07/11/2014 (6:12 am)
@Jesse: As Konrad stated the hit is negligible, I will do some metrics tonight when I get home ... hopefully after I do the Pull Request and will revert back with my findings.I have used the original commercial version before and never really experienced any performance hits.
#7
If so great, where could I download and try it out?
07/11/2014 (6:36 am)
So is this becoming MIT like verve did or not I'm a little confused?If so great, where could I download and try it out?
#8
I unfortunately can't do much here at the office and this will have to wait until I get home.
Hopefully this will eventually get included in the master branch, but for now it will be available as a separate branch ... I think that is the way it works ... head is a little fuzzy from the flu. 8-{
07/11/2014 (6:38 am)
@Kory: I am going to put a Pull Request (if I can figure it out) on to the MIT release. For the meantime I will do a branch of the code and get it working on there for everyone.I unfortunately can't do much here at the office and this will have to wait until I get home.
Hopefully this will eventually get included in the master branch, but for now it will be available as a separate branch ... I think that is the way it works ... head is a little fuzzy from the flu. 8-{
#9
Thanks for your generosity. I look forward to trying it out.
@ Quinton
I just need access to the source code and I can throw together a PR quite quickly as well as review the code and shaders for any improvements/fixes. With that said though, we're not in a huge hurry, maybe it's a good opportunity for you to learn how to do PR's :D
@ Jesse
FPS is a bad measurement of performance. It's not linear so as the FPS gets lower each frame is worth more. A better measurement is MSPF (milliseconds per frame) which measures how long it takes a frame to render. Just take 1000 and divide it by the FPS and you'll get MSPF.
Example:
At 500fps, that's 2 mspf. At 250fps it's 4 mspf. That sounds like a big jump, but it's only 2ms per-frame in the difference. Lets compare say 50 fps and 40 fps:
At 50 fps you're looking at 20 mspf, at 40 fps you're looking at 25 mspf. That's a drop of 10 FPS but an increase of 5 mspf. It takes more to drop the framerate by 10 when you're at 50fps, then it does to drop the framerate by 250 when you're at 500fps.
The summarize what I'm saying: When you're operating at a high FPS it doesn't take much to cut your "performance" in half. As you settle out to a more stable framerate these hits have less impact. So, don't worry about dropping from 500fps to 250fps. Worry about dropping from 60 to 50 as it's a much bigger fall.
07/11/2014 (6:48 am)
@ KonradThanks for your generosity. I look forward to trying it out.
@ Quinton
I just need access to the source code and I can throw together a PR quite quickly as well as review the code and shaders for any improvements/fixes. With that said though, we're not in a huge hurry, maybe it's a good opportunity for you to learn how to do PR's :D
@ Jesse
FPS is a bad measurement of performance. It's not linear so as the FPS gets lower each frame is worth more. A better measurement is MSPF (milliseconds per frame) which measures how long it takes a frame to render. Just take 1000 and divide it by the FPS and you'll get MSPF.
Example:
At 500fps, that's 2 mspf. At 250fps it's 4 mspf. That sounds like a big jump, but it's only 2ms per-frame in the difference. Lets compare say 50 fps and 40 fps:
At 50 fps you're looking at 20 mspf, at 40 fps you're looking at 25 mspf. That's a drop of 10 FPS but an increase of 5 mspf. It takes more to drop the framerate by 10 when you're at 50fps, then it does to drop the framerate by 250 when you're at 500fps.
The summarize what I'm saying: When you're operating at a high FPS it doesn't take much to cut your "performance" in half. As you settle out to a more stable framerate these hits have less impact. So, don't worry about dropping from 500fps to 250fps. Worry about dropping from 60 to 50 as it's a much bigger fall.
#10
I have the powaaaaaaaaaaaaaaaaaaaaaaaaaa!!!!
Fork: help.github.com/articles/fork-a-repo
Pull Request: help.github.com/articles/using-pull-requests
07/11/2014 (6:50 am)
@Andrew: Roger that. I shall accept the challenge and learn how to do Pull Requests ... Google is my friend ... and I am not afraid to use it. 8-}I have the powaaaaaaaaaaaaaaaaaaaaaaaaaa!!!!
Fork: help.github.com/articles/fork-a-repo
Pull Request: help.github.com/articles/using-pull-requests
#11
And thanks to Quinton for rolling it in!
Only took me three tries to get Quinton's name right.... sorry....
07/11/2014 (6:54 am)
Thanks Konrad!And thanks to Quinton for rolling it in!
Only took me three tries to get Quinton's name right.... sorry....
#12
To summarize the process:
1) Make an account on github
2) Fork the Torque3D repo
3) On github make a new branch based on either master or dev branch and call it something related to what you're doing. Like sahara.
4) Clone that repo to your local computer using your favorite git program. I use sourcetree.
5) Apply the changes you're looking to make, be sure to follow the style guidelines and include comments with the code to make it obvious what things are doing.
6) Commit your changes and push them, including a descriptive commit message that makes it clear these are all sahara related changes.
7) On github go to your repo and select your branch. You'll see something along these lines: "This branch is 12 commits ahead and 0 commits behind master" and over to the right it has Compare.
8) Click on Compare, click Edit and make sure you're selecting the development branch of Torque3D repo. Click "Create Pull Request" and fill out the description.
That should be about it. Don't be afraid to try. Worst case scenario you just have to cancel it and make the request again.
07/11/2014 (6:57 am)
@ QuintonTo summarize the process:
1) Make an account on github
2) Fork the Torque3D repo
3) On github make a new branch based on either master or dev branch and call it something related to what you're doing. Like sahara.
4) Clone that repo to your local computer using your favorite git program. I use sourcetree.
5) Apply the changes you're looking to make, be sure to follow the style guidelines and include comments with the code to make it obvious what things are doing.
6) Commit your changes and push them, including a descriptive commit message that makes it clear these are all sahara related changes.
7) On github go to your repo and select your branch. You'll see something along these lines: "This branch is 12 commits ahead and 0 commits behind master" and over to the right it has Compare.
8) Click on Compare, click Edit and make sure you're selecting the development branch of Torque3D repo. Click "Create Pull Request" and fill out the description.
That should be about it. Don't be afraid to try. Worst case scenario you just have to cancel it and make the request again.
#13
@Quinton: Nice.
@Andrew: Thanks for the in-depth explanation on that. Once again, your posting has been enlightening. This gives me a lot to consider while doing my tests. I've not looked so deeply into the fps metric. I've just finished re-compiling each build of Torque I've used and think I've isolated the issue to a single build using Recast.
07/11/2014 (7:01 am)
@Konrad: Special thanks to you for this!@Quinton: Nice.
@Andrew: Thanks for the in-depth explanation on that. Once again, your posting has been enlightening. This gives me a lot to consider while doing my tests. I've not looked so deeply into the fps metric. I've just finished re-compiling each build of Torque I've used and think I've isolated the issue to a single build using Recast.
#14
@Andrew: Roger, thanks. I will tackle this tonight and do what needs to be done to get this in.
@All: What I will do is post a resource as well with the patch as well as the changed files and new files for those that can't wait for the Pull Request.
@Konrad: I take it that this will okay to do now that we have kind of made the announcement. 8-}
07/11/2014 (7:02 am)
@Richard: LOL ... just call me Q ... everyone does. 8-}@Andrew: Roger, thanks. I will tackle this tonight and do what needs to be done to get this in.
@All: What I will do is post a resource as well with the patch as well as the changed files and new files for those that can't wait for the Pull Request.
@Konrad: I take it that this will okay to do now that we have kind of made the announcement. 8-}
#15
They are in essence part of the shaders and materials side of things.
I am thinking of putting them in shaderGen. Comments?
07/11/2014 (7:33 am)
@Andrew: Where would like me to put the "accuFeature.cpp" and "accuFeature.h" files. They currently sit in Engine/source/sahara ... not sure if they should stay there or be put somewhere else.They are in essence part of the shaders and materials side of things.
I am thinking of putting them in shaderGen. Comments?
#16
Before you make the pull request you should put up a working version on github so we can look at it. When I first read of Sahara there was something that occurred to me right away. Accumulation map is per-material so if I put sand as an accumulation map on a statue's material, I can't reuse that statue in a grass level without changing it to a different material. At that point you could just use different diffuse maps with accumulation baked in. The addon makes life easier for sure, but I'm wondering if we could do more with it.
I'm thinking we could make a more flexible system using the original shaders with some modifications to how accumulation map is supplied. What if we made accumulation markers you place in the world editor like spawn points. The marker will have a single material and radius but isn't rendered in game. Any objects within the radius of the accumulation marker that have accumulation enabled would use the marker's material as the accumulation map. This way you could have 1 statue with 1 material in two different biomes (forest, and sand for example) within the same level and have sand accumulation on one and moss accumulation on the other. Maybe later we could explore overlapping markers for double accumulation.
If people are interested I'm confident I could rig this up, I'm quite familiar with shadergen and materials features after the deferred shading work. I just need access to the sahara code to work off of.
07/11/2014 (10:25 am)
Are the classes in accuFeature.cpp inheriting from ShaderFeatureHLSL? If so, they'd probably belong in ShaderFeatureHLSL.cpp. Before you make the pull request you should put up a working version on github so we can look at it. When I first read of Sahara there was something that occurred to me right away. Accumulation map is per-material so if I put sand as an accumulation map on a statue's material, I can't reuse that statue in a grass level without changing it to a different material. At that point you could just use different diffuse maps with accumulation baked in. The addon makes life easier for sure, but I'm wondering if we could do more with it.
I'm thinking we could make a more flexible system using the original shaders with some modifications to how accumulation map is supplied. What if we made accumulation markers you place in the world editor like spawn points. The marker will have a single material and radius but isn't rendered in game. Any objects within the radius of the accumulation marker that have accumulation enabled would use the marker's material as the accumulation map. This way you could have 1 statue with 1 material in two different biomes (forest, and sand for example) within the same level and have sand accumulation on one and moss accumulation on the other. Maybe later we could explore overlapping markers for double accumulation.
If people are interested I'm confident I could rig this up, I'm quite familiar with shadergen and materials features after the deferred shading work. I just need access to the sahara code to work off of.
#17
I am still going to do a resource for those that require it.
Enjoy and thanks to Konrad for letting me do this. You sir are a gentleman and a scholar. 8-}
07/11/2014 (10:40 am)
@Andrew: The current version can be found here: github.com/qdelpeche/Torque3D/tree/sahara ... I am still running some tests and I haven't submitted the Pull Request yet.I am still going to do a resource for those that require it.
Enjoy and thanks to Konrad for letting me do this. You sir are a gentleman and a scholar. 8-}
#18
Quinton, good luck getting up to speed with git. Give us a shout if there's any trouble :). And like Andrew said, do make your work public before PRing it.
Andrew, that sounds exactly like how it should work.
07/11/2014 (10:52 am)
Konrad, that's a wonderful boon to the community! Thanks very much.Quinton, good luck getting up to speed with git. Give us a shout if there's any trouble :). And like Andrew said, do make your work public before PRing it.
Andrew, that sounds exactly like how it should work.
#19
I have also created a resource for the time being, read it here.
Here is a sample image of my running test ... nothing spectacular ... yet ... 8-}
Weeee ... this was fun.
07/11/2014 (11:08 am)
Right, tests finished ... code pushed up to my branch ... I will wait for final feedback before I do the Pull Request.I have also created a resource for the time being, read it here.
Here is a sample image of my running test ... nothing spectacular ... yet ... 8-}
Weeee ... this was fun.
#20
@Daniel: I never had the time to get started with the port myself, so I was happy that Quinton offered his help. I meant to do it for years. Either way, I'm glad it's in a PR now. Maybe Meshloop's next.
07/11/2014 (4:51 pm)
@Andrew: Great idea about how the accumulation map could be passed over to the shader! It makes a lot more sense to use it that way. It's really exciting to hear ideas like yours just a few hours after it's out there. :) Fantastic!@Daniel: I never had the time to get started with the port myself, so I was happy that Quinton offered his help. I meant to do it for years. Either way, I'm glad it's in a PR now. Maybe Meshloop's next.
Torque Owner Jesse Allen
I wanted to post here with a thanks and an inquiry. First off, just thanks for doing this. At every turn, I see Torque being sized up against the competition and it's things like this that make being a part of the Torque community a special treat! Good on you, Quinton!
I was wanting to inquire about performance with this in use. I only ask because recently it's come to my attention that pretty much any source change I've tested is seriously degrading performance. I can start up a Full template room and get over 500fps with a default T3D install. Any source changes whatsoever, and for some reason I've been getting chopped in half to 250ish or less in some cases. Weird right? Anyhow, just wondering about the fps impact putting this stuff to use in MIT.
Cheers, and thanks again for your work Quinton!