The future of the Project Generator
by Daniel Buckmaster · in Torque 3D Professional · 11/28/2014 (4:26 am) · 11 replies
In version 3.6 of T3D, we added support for generating projects to compile the engine with CMake. We did this because CMake is a popular build file generator, and it gives us support for a large variety of build tools without any maintenance burden on us. That is one of the key benefits - we no longer have to manually create templates to support generation of makefiles, VS projects, etc. - that is handled by CMake.
However, as we've discovered, CMake doesn't have a lot of benefits that the Project Generator does. As I see it, the key benefit of the PG at the moment is that you can store its configuration (I'm talking about projectCode.conf), version control it, etc. It allows you to share and persist the configuration of your project - modules, additional files - in a Torque-specific fashion, because the PG knows about Torque, which CMake never can.
If there are any other key features the PG has that CMake doesn't, please let me know. Generating the DLL is not one of these features - Luis tells me we can easily tweak the CMake to reintroduce the DLL.
However, Luis has realised recently that supporting things like SDL on Linux, or LuaJIT, will become very difficult using the current Project Generator - maintaining feature parity with CMake will become a lot of work. For this reason, we want to propose that the current PHP build file generator - what we know as the Project Generator - be deprecated in favour of using CMake to generate build files.
However, we realise and embrace the fact that we need to have a layer above CMake with Torque-specific abilities. For this reason, the Project Generator will remain, but will use CMake as a backend to perform the actual build file generation. So instead of having templates for makefiles and VS projects in PHP, we'll use CMake for all that. But the project Generator will come up with a CMake command and run it automatically, so you don't have to deal with CMake's interface.
What this new Project Generator will look like exactly is uncertain. Will it still be in PHP? How much will it affect the Project Manager? These questions and more are up for discussion. I'm really interested to see how people feel about this. I have some ideas of my own forming, but they're far from mature.
However, as we've discovered, CMake doesn't have a lot of benefits that the Project Generator does. As I see it, the key benefit of the PG at the moment is that you can store its configuration (I'm talking about projectCode.conf), version control it, etc. It allows you to share and persist the configuration of your project - modules, additional files - in a Torque-specific fashion, because the PG knows about Torque, which CMake never can.
If there are any other key features the PG has that CMake doesn't, please let me know. Generating the DLL is not one of these features - Luis tells me we can easily tweak the CMake to reintroduce the DLL.
However, Luis has realised recently that supporting things like SDL on Linux, or LuaJIT, will become very difficult using the current Project Generator - maintaining feature parity with CMake will become a lot of work. For this reason, we want to propose that the current PHP build file generator - what we know as the Project Generator - be deprecated in favour of using CMake to generate build files.
However, we realise and embrace the fact that we need to have a layer above CMake with Torque-specific abilities. For this reason, the Project Generator will remain, but will use CMake as a backend to perform the actual build file generation. So instead of having templates for makefiles and VS projects in PHP, we'll use CMake for all that. But the project Generator will come up with a CMake command and run it automatically, so you don't have to deal with CMake's interface.
What this new Project Generator will look like exactly is uncertain. Will it still be in PHP? How much will it affect the Project Manager? These questions and more are up for discussion. I'm really interested to see how people feel about this. I have some ideas of my own forming, but they're far from mature.
About the author
Studying mechatronic engineering and computer science at the University of Sydney. Game development is probably my most time-consuming hobby!
#2
so you have to download that cmake tool now to build torque? don't fix when it's not broken just like what they say. for me and i'm sure others who are not that tech aware, it's good as it is.. keep it simple
11/28/2014 (5:32 am)
Daniel, im against that cmake thing imo. the engine will look more technically challenging even though you package it right. so you have to download that cmake tool now to build torque? don't fix when it's not broken just like what they say. for me and i'm sure others who are not that tech aware, it's good as it is.. keep it simple
#3
11/28/2014 (5:37 am)
Yeah but CMake can be shipped with T3D. The whole point of the project manager is so you have a nice GUI interface to create projects and not care what system it is using under the hood.
#4
Just to clarify, i'm all for CMake BUT only if it can be made to work using the more friendly interface like the current project manager. For the more technical users that are comfortable using CMake directly than sweet do that but we don't to want alienate people that are not comfortable using it.
11/28/2014 (5:41 am)
Oh sh*t hang on i didn't read dans post properly, sorry lol. He wants to get rid of the project manager entirely?? If so, i agree that is not a good idea.Just to clarify, i'm all for CMake BUT only if it can be made to work using the more friendly interface like the current project manager. For the more technical users that are comfortable using CMake directly than sweet do that but we don't to want alienate people that are not comfortable using it.
#6
As for keeping projectGenerator alive, I prefer it over CMake, and I will continue to keep it operational as I've already had to do for TGE. Especially considering CMake requires its files all over the damn place in order to build Torque while projectGenerator it's all in one small directory structure that makes it easy to maintain and keep track of.
11/28/2014 (8:37 am)
CMake will not help you with libSDL on Linux either. No idea about LuaJIT since I haven't looked into it. As for libSDL (especially 2.0) will require the user to build the library themselves to fit their Linux distribution or the user's system environment as libSDL has just too many configuration settings to have a one-config-fits-all that could be used by projectGenerator nor CMake.As for keeping projectGenerator alive, I prefer it over CMake, and I will continue to keep it operational as I've already had to do for TGE. Especially considering CMake requires its files all over the damn place in order to build Torque while projectGenerator it's all in one small directory structure that makes it easy to maintain and keep track of.
#7
About use CMake. Working on external libraries i have find CMake more easy to handle special compilation necesities for each library.
Another important thing about use CMake is we don't need to mantain this part of code. My "personal" idea is try to reduce size of T3D code and our maintenance work replacing some code to external libraries.
We can use SDL2 for window handle and input c++ libraries for containers, thread, smart pointers... Raknet for networking Assimp for handle external 3d formats ...
Some clarifications:
The idea is preserve ProjectManager GUI Cmake are not going to increase the dificulty for users. CMake can reduce our maintenance work and reduce size of T3D CMake are very common used on open source projects, can be more easy to new users. We can distribute CMake binaries with T3D, we do the same with php5 binaries now. CMake can make compilation of external libraries more easy CMake files are only on 1 directory. See here. CMake handle libraries/modules very clean and easy. See here.
11/28/2014 (11:04 am)
About Project Manager GUI. The idea is preserve this GUI, but use CMake instead of php for generate solution files. This change are not going to increase the dificulty for users.About use CMake. Working on external libraries i have find CMake more easy to handle special compilation necesities for each library.
Another important thing about use CMake is we don't need to mantain this part of code. My "personal" idea is try to reduce size of T3D code and our maintenance work replacing some code to external libraries.
Some clarifications:
#8
Maybe we should also talk about the future of the Project Manager if people are interested - I figured on not talking about it in this thread so as exactly not to cause confusion between the Project Generator and Project Manager :P.
TRON, have you looked at the current CMake implementation at all? It's all inside Tools/CMake, plus one file in the root directory. Obviously libraries will have their own CMake setup, but we don't need to worry about them.
The thing I dislike the most about CMake is its awful configuration language. Ideally we'll hide most of this from the user, too. I've been hoping we can work with JSON- or XML-based configuration files, which would replace the existing projectCode.conf and friends, to let you specify which source files to include, exclude, etc. Modules could use the same format.
11/28/2014 (12:45 pm)
Right, where did the idea of scrapping the Project Manager come from? I've been saying everywhere, often, that I have plans to expand the Project Manager's functionality and usefulness, and didn't even mention it in this post. What about this is unclear?Quote:But the project Generator will come up with a CMake command and run it automatically, so you don't have to deal with CMake's interface.
Maybe we should also talk about the future of the Project Manager if people are interested - I figured on not talking about it in this thread so as exactly not to cause confusion between the Project Generator and Project Manager :P.
TRON, have you looked at the current CMake implementation at all? It's all inside Tools/CMake, plus one file in the root directory. Obviously libraries will have their own CMake setup, but we don't need to worry about them.
The thing I dislike the most about CMake is its awful configuration language. Ideally we'll hide most of this from the user, too. I've been hoping we can work with JSON- or XML-based configuration files, which would replace the existing projectCode.conf and friends, to let you specify which source files to include, exclude, etc. Modules could use the same format.
#9
I would be cautious about assuming "we don't need to maintain this part of code" since cmake itself has changed quite a bit over the past few years. Nothing based on a moving target is maintenance free.
I haven't seen an example of a project where cmake has been bundled into the SDK. Would be interested to see if this has been done before.
At the end of the day for me, if it generates correct VC++ projects with proper build configurations then no problem.
11/28/2014 (2:34 pm)
Personally I think the project generator, despite being a bit feature locked (though slightly improved in tgemit), works fine. Usually I only use it once to generate the basic projects and leave everything.I would be cautious about assuming "we don't need to maintain this part of code" since cmake itself has changed quite a bit over the past few years. Nothing based on a moving target is maintenance free.
I haven't seen an example of a project where cmake has been bundled into the SDK. Would be interested to see if this has been done before.
At the end of the day for me, if it generates correct VC++ projects with proper build configurations then no problem.
#10
Well, I take that back it needs to be able to generate switchable 32/64 bit builds. I don't want to have to run the damn thing everytime I switch between 32 and 64 bit
11/28/2014 (2:34 pm)
As long as it generates the DLL and it's easy to use I'm fine.Well, I take that back it needs to be able to generate switchable 32/64 bit builds. I don't want to have to run the damn thing everytime I switch between 32 and 64 bit
#11
Vince, that's definitely a problem. As we wouldn't handle solution generation directly, we'd rely on CMake itself to actually create the solutions, and they don't support mixed 32/64 solutions. However, what I've been thinking for cases like this is that projects should have a number of configurations. You can then generate a solution for each of those configurations. So in your case, you'd have two VS solutions, one for 32 and one for 64, which you could alt-tab between if you need them both at the same time. You'd only need to run the project generator when you change the source files that are in your project, and then it would regenerate both projects. The same thing could handle dedicated server config, etc. Basically, like the current torque3D_dedicated.conf, for example.
11/28/2014 (4:47 pm)
Quote:I would be cautious about assuming "we don't need to maintain this part of code" since cmake itself has changed quite a bit over the past few years. Nothing based on a moving target is maintenance free.That's true. Luis and I are both referring to actually supporting various backends (VS projects, makefiles, etc.), which we don't have to maintain - but we would have to maintain our own CMake code, yes.
Vince, that's definitely a problem. As we wouldn't handle solution generation directly, we'd rely on CMake itself to actually create the solutions, and they don't support mixed 32/64 solutions. However, what I've been thinking for cases like this is that projects should have a number of configurations. You can then generate a solution for each of those configurations. So in your case, you'd have two VS solutions, one for 32 and one for 64, which you could alt-tab between if you need them both at the same time. You'd only need to run the project generator when you change the source files that are in your project, and then it would regenerate both projects. The same thing could handle dedicated server config, etc. Basically, like the current torque3D_dedicated.conf, for example.
Timmy01
Thinking about generic T3D, is your average user even aware of what the project manager does under the hood and do they even care? (i hope that doesn't sound condescending) I see no problem dumping the old system and using CMake under the hood, as long as it generates the same build files i think it's fine and actually a better and more flexible option. For example it would allow more options in the project manager interface, you could easily add a drop down list for common compilers, things like that.