Torque3D's Import System
by raa brubb · in Torque 3D Professional · 05/05/2014 (2:39 pm) · 75 replies
Hey all,
I've been doing some research about OpenGL, Assimp, and other things. I would like to implement Assimp (even though that would be a while from now) into Torque3D. My only concern was how does Torque3D's import system work? Does it only support DTS and import DAEs by conversion.
I had previously brought up this idea in the forums, but I seriously think it should be implemented into Torque3D 4.0. I'm not pushing anything, but this would seriously fix the problem with importing. The only importing supported by Torque3D is DAE and FBX (using a community add-on). This could be fixed using Assimp. Not to mention Assimp has an open source viewer as well. What do you think? Could this be easily implemented or totally impossible?
I've been doing some research about OpenGL, Assimp, and other things. I would like to implement Assimp (even though that would be a while from now) into Torque3D. My only concern was how does Torque3D's import system work? Does it only support DTS and import DAEs by conversion.
I had previously brought up this idea in the forums, but I seriously think it should be implemented into Torque3D 4.0. I'm not pushing anything, but this would seriously fix the problem with importing. The only importing supported by Torque3D is DAE and FBX (using a community add-on). This could be fixed using Assimp. Not to mention Assimp has an open source viewer as well. What do you think? Could this be easily implemented or totally impossible?
About the author
#22
Also, boost is only used for multithreading in assimp and it won't matter after the models are loaded. Assimp loads the model into it's generic format, that gets loaded into a TSShape which is then saved back out as a cached.dts file. During map loading and what not in a production game assimp shouldn't even be used since your game will ship with dts files. Single threaded is perfectly fine for this application.
05/10/2014 (1:18 pm)
Boost is not horrible in terms of programming, it's very well written and works very well. It's just to me it's always felt like a huge toolbox on the top shelf and when you go to get one tool from it the whole thing comes crashing down on your head. Next time you to go to use it you think twice about reaching for it and eventually just stop altogether and buy new tools. Also, boost is only used for multithreading in assimp and it won't matter after the models are loaded. Assimp loads the model into it's generic format, that gets loaded into a TSShape which is then saved back out as a cached.dts file. During map loading and what not in a production game assimp shouldn't even be used since your game will ship with dts files. Single threaded is perfectly fine for this application.
#23
05/10/2014 (4:50 pm)
Oh, OK thanks! Also, I was wondering if there were any online tutorials you could suggest so I could help more with T3D. Looks like you know a lot! Good luck tomorrow on Assimp. 
#24
05/11/2014 (11:53 am)
You could retrofit it with OpenMP, but that's another can of worms because it must be supported by your compiler directly. Or update to C++ 11, but that's a third can of worms....
#25
I have less than a year of experience with Torque, I've just spent a lot of time ripping things apart and putting them back in plenty of big projects before, so the code base doesn't scare me.
@Richard
Too many worms for me. Luckily the library seems to perform well without multi-threading.
Progress Update:
I put in my time today and I'm quite happy with the progress. I've basically written all the groundwork for loading the data in and it's semi-functioning. No crashes, errors, warnings, etc. I sorted them all out. It navigates the nodes and builds the shape hierarchy. It also creates all the meshes along the way and builds the shape. Here is a screenshot from a .blend file loaded in:
I haven't done anything on materials or UVs yet so the shape doesn't display at the moment but it has the correct number of polys for each mesh and hasn't produced any errors building the shape after I feed it the verts, indices, and primitive data.
05/11/2014 (12:26 pm)
@raaI have less than a year of experience with Torque, I've just spent a lot of time ripping things apart and putting them back in plenty of big projects before, so the code base doesn't scare me.
@Richard
Too many worms for me. Luckily the library seems to perform well without multi-threading.
Progress Update:
I put in my time today and I'm quite happy with the progress. I've basically written all the groundwork for loading the data in and it's semi-functioning. No crashes, errors, warnings, etc. I sorted them all out. It navigates the nodes and builds the shape hierarchy. It also creates all the meshes along the way and builds the shape. Here is a screenshot from a .blend file loaded in:
I haven't done anything on materials or UVs yet so the shape doesn't display at the moment but it has the correct number of polys for each mesh and hasn't produced any errors building the shape after I feed it the verts, indices, and primitive data.
#26
05/11/2014 (3:40 pm)
Holy crap Andrew, that's fantastic. What's it doing with the lamp, camera, etc.?
#27
This actually seems like an easy library. Its great to hear its actually do-able. You should put a pull request after you finish into the Torque3D Git for more file formats. This is freaking amazing.
05/11/2014 (4:12 pm)
Wow, that amazing. You did this pretty fast. So how would it deal with like particles or oceans that may be included in a .blend file?This actually seems like an easy library. Its great to hear its actually do-able. You should put a pull request after you finish into the Torque3D Git for more file formats. This is freaking amazing.
#28
05/11/2014 (5:30 pm)
Great work Andrew..I'm speechless :)
#29
Those are just nodes in the blender scene. They actually show up as a light and a camera in blender. They're stored in the file so they're loaded in but they don't contain any mesh data ( verts, normals, etc ) so they don't do anything but show up in the node list.
@ Raa
Features like particles, oceans, etc are not standard across model formats so assimp filters them out.
Progress Update:
I wasn't going to work on it anymore today but I knew it wouldn't be much to add materials in, so I did. And, we have visibility!
On the left is dwarf.x, and on the right is 4cube.blend. I haven't got node transformations working yet so the 4 cubes appear as 1 since they're all in the same spot. Also, no UV data yet so they're all stuck as orange for now.
05/11/2014 (9:52 pm)
@ DanielThose are just nodes in the blender scene. They actually show up as a light and a camera in blender. They're stored in the file so they're loaded in but they don't contain any mesh data ( verts, normals, etc ) so they don't do anything but show up in the node list.
@ Raa
Features like particles, oceans, etc are not standard across model formats so assimp filters them out.
Progress Update:
I wasn't going to work on it anymore today but I knew it wouldn't be much to add materials in, so I did. And, we have visibility!
On the left is dwarf.x, and on the right is 4cube.blend. I haven't got node transformations working yet so the 4 cubes appear as 1 since they're all in the same spot. Also, no UV data yet so they're all stuck as orange for now.
#30
Also, I was thinking about the multi-threading option being taken out and realized that the only time it might be used is if the person decides to reload the models when loading the mission. Its probably unsmart to reload it each time, but would it effect performance?
05/11/2014 (11:20 pm)
AMAZING! I can't believe its being finished so fast!Also, I was thinking about the multi-threading option being taken out and realized that the only time it might be used is if the person decides to reload the models when loading the mission. Its probably unsmart to reload it each time, but would it effect performance?
#31
05/12/2014 (12:19 am)
I'll pull this down and help out when I can. This sounds like a 3.7 kind of change? Unless you want to submit a basic PR before next Monday, which is the deadline for 3.6. 
#32
05/12/2014 (12:45 am)
@Andrew, great work :D 
#33
05/12/2014 (4:23 am)
Looking good! Can't wait for the node transforms, animations, materials, and everything else applicable to be implemented ;) 
#34
05/12/2014 (1:24 pm)
Nice work
#35
Once a model is imported into torque its saved as dts. So, after you load in your model and goes through assimp, the next time you load the model it will load the dts and not use assimp at all. Also, I've noticed no difference in the time it takes to import a model through assimp VS via regular collada. I'm sure it's measurable on the ms scale, but not noticable to me. Actually it may be an interesting experiment to use assimp to import collada into torque and see which performs better.
@ Dan
The biggest obstacle I have right now is getting assimp to compile as one of torque's libraries. If we can cross that off the list I'd say we're golden. I'm running off an install of the SDK rather than compiling it as part of the engine.
Progress Update:
UVs are done, and node transformations. My 4cube.blend is still not displaying 4 cubes though. It has the right sized bounding box and node positions so I suspect its a material issue. Dwarf is working great though:
I'm going to treat this project as two different pieces. The first being static shapes, the second being animations. Assimp supports BVH and CSM animation files so it's a whole other thing to get into. I'm rounding the bend on static shape functionality. I've already started adding the different formats to the editors so you can directly import, and it's working well. There's just a few more things to do before the static side is ready for testing.
05/12/2014 (2:39 pm)
@ RaaOnce a model is imported into torque its saved as dts. So, after you load in your model and goes through assimp, the next time you load the model it will load the dts and not use assimp at all. Also, I've noticed no difference in the time it takes to import a model through assimp VS via regular collada. I'm sure it's measurable on the ms scale, but not noticable to me. Actually it may be an interesting experiment to use assimp to import collada into torque and see which performs better.
@ Dan
The biggest obstacle I have right now is getting assimp to compile as one of torque's libraries. If we can cross that off the list I'd say we're golden. I'm running off an install of the SDK rather than compiling it as part of the engine.
Progress Update:
UVs are done, and node transformations. My 4cube.blend is still not displaying 4 cubes though. It has the right sized bounding box and node positions so I suspect its a material issue. Dwarf is working great though:
I'm going to treat this project as two different pieces. The first being static shapes, the second being animations. Assimp supports BVH and CSM animation files so it's a whole other thing to get into. I'm rounding the bend on static shape functionality. I've already started adding the different formats to the editors so you can directly import, and it's working well. There's just a few more things to do before the static side is ready for testing.
#36
05/12/2014 (2:52 pm)
:) 
#38
05/12/2014 (3:50 pm)
Andrew, you are a coding animal! We are very lucky to have you as part of our community, thanks for the great work you're doing!
#39
This is a busy month for me, but I really want to get this pushed in for 3.6. I know what we can do to cheat the issue deadline... #642
05/12/2014 (4:55 pm)
Excellent. Have you tried, for example, an FBX from the Unity store if there are any free ones? Being able to import FBX statics would be a fantastic MVP to integrate this.This is a busy month for me, but I really want to get this pushed in for 3.6. I know what we can do to cheat the issue deadline... #642
#40
05/12/2014 (5:11 pm)
I could supply a .FBX from the unity store that is free or what ever. Just let me know and I'll upload it somewhere! 
raa brubb