Abandoning Torque after six months of development
by Nir Ziso · in Torque Game Builder · 01/25/2007 (3:20 am) · 34 replies
Hello all,
Our company has been developing a card game with Torque for the last six months. Everything was going well until in QA on Pentium 4 machines something very disturbing happened. Torque is taking all available CPU in order to achieve maximum FPS and the PC comes to 100% CPU. Furthermore all animations become jittery or stuck and the mouse becomes jumpy or non responsive.
We tried the forums for answers or hints to no avail. We even have the C++ sources but discovering such a problem late in the development and shortly before the release is not quite possible.
Graphics quality is not the issue either. We run the TGB editor on this P4 without any animations or the game without rendering anything but the background and still the same problems occur. We also downloaded other games like "Puzzle Poker" or "Saints and Sinners Bowling" from Large Animal (By the way great job guys your art and concept rocks!) but the CPU results where the same.
Moreover OpenGL issues are a headache when you get to the end user. Telling users to update the drivers of their graphics card is inconceivable, especially when our competitors who wrote their games in pure MFC don't ask that of them. We can't go full screen either. Again our competitors don't ask that from their users and neither can we.
When you bring a product to the end user if you want him to buy it, giving answers such as "Play in full screen" or "Update your graphics card drivers" are unacceptable.
Finally I must point out that while compiling the C++ sources we discovered that the warning level on the Visual Studio compiler was set to 1. This means "Don't bother me with your warnings" (more or less). When changing it to level 3 we got 1500 warnings. After 3 days we are down to 300 and stopping the effort. Again the answers to questions posted to the developers where unacceptable. Cleaning up your code from warnings is not "obsessive compulsive" or "tedious". It is best practice.
So for now we abandon Torque and plan to rewrite the whole game in Flash or Director.
When the product becomes mature enough and the "obsessive compulsive" and "tedious" tasks of warnings cleanup are done with in the "nearish" future (GG developers who answered to me as such know what I am talking about) we may reconsider trying our luck again with Torque.
We are all here are C++ developers but we BOUGHT a game engine not an open source product. We want to extend the engine not fix it or change it and especially not getting into the task of having to "adjust the engine's amount of CPU usage" (again GG developers answer) and have to merge it again in the next engine version and try to tackle any new CPU killers in the engine. After all we want to develop a game not a game engine.
Oh and by the way you can flame me all you want for this post but facts are facts and unprofessional answers were the last drop.
Best Regards and good luck with your games everybody.
New Reign R&D team.
Our company has been developing a card game with Torque for the last six months. Everything was going well until in QA on Pentium 4 machines something very disturbing happened. Torque is taking all available CPU in order to achieve maximum FPS and the PC comes to 100% CPU. Furthermore all animations become jittery or stuck and the mouse becomes jumpy or non responsive.
We tried the forums for answers or hints to no avail. We even have the C++ sources but discovering such a problem late in the development and shortly before the release is not quite possible.
Graphics quality is not the issue either. We run the TGB editor on this P4 without any animations or the game without rendering anything but the background and still the same problems occur. We also downloaded other games like "Puzzle Poker" or "Saints and Sinners Bowling" from Large Animal (By the way great job guys your art and concept rocks!) but the CPU results where the same.
Moreover OpenGL issues are a headache when you get to the end user. Telling users to update the drivers of their graphics card is inconceivable, especially when our competitors who wrote their games in pure MFC don't ask that of them. We can't go full screen either. Again our competitors don't ask that from their users and neither can we.
When you bring a product to the end user if you want him to buy it, giving answers such as "Play in full screen" or "Update your graphics card drivers" are unacceptable.
Finally I must point out that while compiling the C++ sources we discovered that the warning level on the Visual Studio compiler was set to 1. This means "Don't bother me with your warnings" (more or less). When changing it to level 3 we got 1500 warnings. After 3 days we are down to 300 and stopping the effort. Again the answers to questions posted to the developers where unacceptable. Cleaning up your code from warnings is not "obsessive compulsive" or "tedious". It is best practice.
So for now we abandon Torque and plan to rewrite the whole game in Flash or Director.
When the product becomes mature enough and the "obsessive compulsive" and "tedious" tasks of warnings cleanup are done with in the "nearish" future (GG developers who answered to me as such know what I am talking about) we may reconsider trying our luck again with Torque.
We are all here are C++ developers but we BOUGHT a game engine not an open source product. We want to extend the engine not fix it or change it and especially not getting into the task of having to "adjust the engine's amount of CPU usage" (again GG developers answer) and have to merge it again in the next engine version and try to tackle any new CPU killers in the engine. After all we want to develop a game not a game engine.
Oh and by the way you can flame me all you want for this post but facts are facts and unprofessional answers were the last drop.
Best Regards and good luck with your games everybody.
New Reign R&D team.
#2
I don't disagree that engine problems exist and that they should be fixed, but you must put the whole thing in perspective. 1 month lost still sucks, but it isn't 6. Know what I mean?
Having said that, if you took care of 900+ warnings, it would be incredibly beneficial to the community as a whole to release that as a resource. Often times Indie developers can't afford to bounce from engine to engine, so we get stuck with one (good or bad) and have to make the best of it. Sure, you'd be helping GG on the face, but who you're really helping overall and in the long run are other developers.
01/25/2007 (8:02 am)
It sucks to feel you have wasted time, especially half a year of time, but I can't help but think it's being over-exaggurated. If you think about it, you worked out all the gameplay and art over those six months. You can reuse all the art in flash and programming in actionscript, which is a C like language, wouldn't be that difficult of a port. I would say a 6 month project should go the way of 1 month to port to flash.I don't disagree that engine problems exist and that they should be fixed, but you must put the whole thing in perspective. 1 month lost still sucks, but it isn't 6. Know what I mean?
Having said that, if you took care of 900+ warnings, it would be incredibly beneficial to the community as a whole to release that as a resource. Often times Indie developers can't afford to bounce from engine to engine, so we get stuck with one (good or bad) and have to make the best of it. Sure, you'd be helping GG on the face, but who you're really helping overall and in the long run are other developers.
#3
I see the warnings are worrisome but negligible. I deal with software that has licensing which makes Unreal 3 seem cheap and it is far from warning free. And sometimes they can bite you, especially if your compilation options or linking options work differently on your system than the company's test labs.
But the other two issues you mention, full screen and drivers. Those are two points which you consider deal breakers. I'm interested to know how they did not break the deal for six months. I think it will help people who come here and read this later gain perspective on how issues are dealt with in a business environment.
Again, as I noted in your previous topic, good luck in your endeavors.
01/25/2007 (8:09 am)
I think a postmortem would be a good idea since I am unclear how the project went on for 6 months without these being critical issues for the development staff. Especially if a shift to Director or Flash will alleviate them (and as I agreed with Ward in the other thread, I would not recommend Director).I see the warnings are worrisome but negligible. I deal with software that has licensing which makes Unreal 3 seem cheap and it is far from warning free. And sometimes they can bite you, especially if your compilation options or linking options work differently on your system than the company's test labs.
But the other two issues you mention, full screen and drivers. Those are two points which you consider deal breakers. I'm interested to know how they did not break the deal for six months. I think it will help people who come here and read this later gain perspective on how issues are dealt with in a business environment.
Again, as I noted in your previous topic, good luck in your endeavors.
#4
01/25/2007 (8:11 am)
Jonathon you are right about the porting time of 1 month. This is our estimate as well (thank God we know Flash) but consider that our investor wants us up and running on February 1st... and we can't afford the delay. We will have to run one month with Torque, rewrite and update.
#5
Here is a short postmortem.
It all started when after QAing the game client and game server for 5 months on new machines we decided to birng in some reasonable P4s (mid December). Since then we have two guys trying to tackle the problem.
So in the beginning we stumbled upon some older ATI cards with OpenGL problems. Then we saw the CPU problem (the dev machines are Dual Cores no problem there). In short when you develop your game find a relative or friend with a low end machine (although calling a P4 with 1GB Ram low end sounds weird) and see what happens.
BTW the warnings started poping when we started poking around the engine to fix the CPU problem. While code reviewing we stumbled upon code that should have produced warnings. So we checked the compiler settings and discovered the low warning level. As I said before we want to make a game not a game engine.
01/25/2007 (8:21 am)
David,Here is a short postmortem.
It all started when after QAing the game client and game server for 5 months on new machines we decided to birng in some reasonable P4s (mid December). Since then we have two guys trying to tackle the problem.
So in the beginning we stumbled upon some older ATI cards with OpenGL problems. Then we saw the CPU problem (the dev machines are Dual Cores no problem there). In short when you develop your game find a relative or friend with a low end machine (although calling a P4 with 1GB Ram low end sounds weird) and see what happens.
BTW the warnings started poping when we started poking around the engine to fix the CPU problem. While code reviewing we stumbled upon code that should have produced warnings. So we checked the compiler settings and discovered the low warning level. As I said before we want to make a game not a game engine.
#6
Good write-up. Thanks.
01/25/2007 (8:28 am)
On low-end machines, I have several that I can plug and unplug as need be. And since my target market is not an uber high-end machine or graphics card, I make use of them. I also make use of profiling tools on them to try to narrow the bottlenecks.Good write-up. Thanks.
#8
First of all I'd like to personally apologize to you and your team Nir, as it seems you have not received the help you were looking for previously without having to escalate your problems to this level. To that end, I have failed you, and for that I'm sorry.
Secondly I would like to be honest with you and say that I very much disprove of your manipulation of my words in order to bring more attention to your problems on these forums. You took my attempt at help and twisted it to sound as if I was badgering you or your company.
However, what I said in that email remains true, the compiler warnings with TGB fall way, way down on my radar when it comes to things that I need to give attention to. I'm sorry if this is not what you want to hear, but you need to understand that you are one of many people that I work my tail off to provide software to. When it comes to making a decision about whether to spend a day ridding code of compiler warnings or working on something that 20 community members are having problems with, I'm obviously going to opt for the latter.
Furthermore I assure you we have looked at the warning count and the nature of the warnings in Torque, and I believe Tom Bampton hit it on the head as to how it ended up like this in the thread you posted earlier. We are, as I stated in my email nearing a juncture in our development processes that makes introducing that many changes in the immediate term, a nightmare inducing thing for people that are trying to perform merges from any of our internal projects. We are not ignoring this issue, just reserving the right to resolve it at the appropriate time.
All of this said, I did want to let you know that there is hope for your processor usage problems.
By default TGB will in fact use as much processor as it can get it's hands on, because well, a lot of games want this. But we do understand that people want to be able to control this, which is why we offer a simple preference that you may set that will cause our simulation of time to advance at a different rate.
By default this preference is set to 0 which means that Torque will attempt to advance time as often as possible. If you are building a game (take solitaire for example) that does not need to do a whole bunch of time updates as fast as possible you can set this value to any number of milliseconds.
This will be a tweak-able thing that you'll want to fit right to your project, I find that usage goes down to an acceptable (30% CPU on my 1.6ghz PentiumM) while the game is in focus when I set this value to 10.
Note since this pref does control how often Torque tries to advance time, if you set it too high you will get unpredictable results.
This pref will allow you to specify how long the game will 'sleep' on the system every time it runs through a game process loop. By setting this higher you will see the CPU usage drop when the window is not focused. Much like the above pref, you should tweak this to your needs and experimentation will set you free!
I truly do hope this helps you
Thanks for reading this, once again please accept my apologies for the undoubted frustration this has been causing you. I responded to your thread as soon as it was brought to my attention and was going to get back to your thread about processor usage today. Turns out this might be useful for not only you but many people!
Best Regards,
-Justin
01/25/2007 (6:29 pm)
Hum, how to address this topic...First of all I'd like to personally apologize to you and your team Nir, as it seems you have not received the help you were looking for previously without having to escalate your problems to this level. To that end, I have failed you, and for that I'm sorry.
Secondly I would like to be honest with you and say that I very much disprove of your manipulation of my words in order to bring more attention to your problems on these forums. You took my attempt at help and twisted it to sound as if I was badgering you or your company.
However, what I said in that email remains true, the compiler warnings with TGB fall way, way down on my radar when it comes to things that I need to give attention to. I'm sorry if this is not what you want to hear, but you need to understand that you are one of many people that I work my tail off to provide software to. When it comes to making a decision about whether to spend a day ridding code of compiler warnings or working on something that 20 community members are having problems with, I'm obviously going to opt for the latter.
Furthermore I assure you we have looked at the warning count and the nature of the warnings in Torque, and I believe Tom Bampton hit it on the head as to how it ended up like this in the thread you posted earlier. We are, as I stated in my email nearing a juncture in our development processes that makes introducing that many changes in the immediate term, a nightmare inducing thing for people that are trying to perform merges from any of our internal projects. We are not ignoring this issue, just reserving the right to resolve it at the appropriate time.
How to [b]Reduce Processor usage[/b] of a Game that doesn't need it
All of this said, I did want to let you know that there is hope for your processor usage problems.
By default TGB will in fact use as much processor as it can get it's hands on, because well, a lot of games want this. But we do understand that people want to be able to control this, which is why we offer a simple preference that you may set that will cause our simulation of time to advance at a different rate.
$Pref::TimeManagerProcessInterval- How often Torque tries to process time events (Defaults to Constantly)
By default this preference is set to 0 which means that Torque will attempt to advance time as often as possible. If you are building a game (take solitaire for example) that does not need to do a whole bunch of time updates as fast as possible you can set this value to any number of milliseconds.
This will be a tweak-able thing that you'll want to fit right to your project, I find that usage goes down to an acceptable (30% CPU on my 1.6ghz PentiumM) while the game is in focus when I set this value to 10.
Note since this pref does control how often Torque tries to advance time, if you set it too high you will get unpredictable results.
$Pref::BackgroundSleepTime- How long Torque will sleep when the game is not in focus on the computer (Defaults to 3000ms)
This pref will allow you to specify how long the game will 'sleep' on the system every time it runs through a game process loop. By setting this higher you will see the CPU usage drop when the window is not focused. Much like the above pref, you should tweak this to your needs and experimentation will set you free!
I truly do hope this helps you
Thanks for reading this, once again please accept my apologies for the undoubted frustration this has been causing you. I responded to your thread as soon as it was brought to my attention and was going to get back to your thread about processor usage today. Turns out this might be useful for not only you but many people!
Best Regards,
-Justin
#9
$Pref::TimeManagerProcessInterval
$Pref::BackgroundSleepTime
Thank you a thousand times for that info!!!
01/26/2007 (2:20 am)
@Justin DuJardin$Pref::TimeManagerProcessInterval
$Pref::BackgroundSleepTime
Thank you a thousand times for that info!!!
#10
Hi, your post really caught my attention. Where I work, we've been developing similar games with Flash for several months, and are now considering migrating to Torque. We are going to be doing some extensive testing of Flash to see if it can handle our needs. If it can't we will need to switch, as we are dealing with a very rigid deadline.
I've discussed it with my supervisor, and he agrees, that if we make the switch to Torque, and you switch to Flash, perhaps we can make some sort of exchange. That way the last six months wont be a waste for either of us.
Like I said, the decision hasn't been made yet (I'll know today or Monday), but this could be a factor in that decision.
01/26/2007 (6:25 am)
Mr Ziso,Hi, your post really caught my attention. Where I work, we've been developing similar games with Flash for several months, and are now considering migrating to Torque. We are going to be doing some extensive testing of Flash to see if it can handle our needs. If it can't we will need to switch, as we are dealing with a very rigid deadline.
I've discussed it with my supervisor, and he agrees, that if we make the switch to Torque, and you switch to Flash, perhaps we can make some sort of exchange. That way the last six months wont be a waste for either of us.
Like I said, the decision hasn't been made yet (I'll know today or Monday), but this could be a factor in that decision.
#11
We knew about these two variables but they helped us only on high end machines, the dual core ones. On the low end P4s our one and only one animation, a card being dropped on the table(top down view of the table), was jittery due to these variables. I played with them as much as I could. They either don't have any effect (below 10ms TimeManagerProcessInterval) or have a baf effect. Again it may be privy to our game but alas the frustration and time spent is too much. And for what... a standard 4 player game card. No sounds yet, no special effects, nothing.
For the rest of the stuff you mention I answered to you directly by mail.
01/28/2007 (2:21 am)
@Justin We knew about these two variables but they helped us only on high end machines, the dual core ones. On the low end P4s our one and only one animation, a card being dropped on the table(top down view of the table), was jittery due to these variables. I played with them as much as I could. They either don't have any effect (below 10ms TimeManagerProcessInterval) or have a baf effect. Again it may be privy to our game but alas the frustration and time spent is too much. And for what... a standard 4 player game card. No sounds yet, no special effects, nothing.
For the rest of the stuff you mention I answered to you directly by mail.
#12
01/29/2007 (9:07 am)
What does the profiler say is taking up all the time?
#13
If I put the profiler on.... the whole game comes down on its knees even on a dual core.
I couldn't work with it. I just commented out the #define in the source code.
01/29/2007 (9:32 am)
@PatIf I put the profiler on.... the whole game comes down on its knees even on a dual core.
I couldn't work with it. I just commented out the #define in the source code.
#14
01/29/2007 (9:45 am)
The profiler is the tool that you need to solve that problem. If you don't know where the problem is, you have very little chance of solving it. I am very confused as to why you would spend the time to fix hundreds of compiler warnings which may-or-may-not relate to your performance issue, and not take the time to use the profiler.
#15
We tried fixing the CPU problem. We spend the time for it. We even think we identified the problem in the windows message queue handling... We tried to invetsigate more, broke the engine 3 times, reverted(SVN) and tried again. Looming release dictated that we stop and move on. As I said we are making a game not a game engine.
01/29/2007 (9:50 am)
The warnings is one issue and the CPU another.We tried fixing the CPU problem. We spend the time for it. We even think we identified the problem in the windows message queue handling... We tried to invetsigate more, broke the engine 3 times, reverted(SVN) and tried again. Looming release dictated that we stop and move on. As I said we are making a game not a game engine.
#16
I'm with Pat on the profiler issue. That's the first thing that I use when I find a CPU-numbing crawl. That's SOP for any C/C++ programmer, AFAIK. IDE's are workflow-friendly. Debuggers and profilers are necessities.
01/29/2007 (10:19 am)
That is an interesting mantra considering you are modifying an existing game engine, not making it from scratch.I'm with Pat on the profiler issue. That's the first thing that I use when I find a CPU-numbing crawl. That's SOP for any C/C++ programmer, AFAIK. IDE's are workflow-friendly. Debuggers and profilers are necessities.
#17
01/29/2007 (10:57 am)
I'd have to chime in on that as well because Profiler is a 'must use'. Warnings don't tend to cause cpu intesive opperations, so I wouldn't put too much weight on that. CPU consumption is what profiler is going to help you figure out. It'll pinpoint what is exactly using so much cpu so you know for sure and we can help you debug it from there.
#18
01/29/2007 (11:03 am)
Let me say that again. The profiler brought my dual core on it's knees. Nothing could move, the game could hardly move and I didn't even start it. I tried thye profiler long before I gave up.
#19
01/29/2007 (11:17 am)
Have you tried the profiler on a non-dual core machine?
#20
Looking back at the forums, I see no reference to your problem except the thread about CPU usage near 50% at all times, which is perfectly normal for Torque. Last month you posted a thread here regarding problems using 'schedule' to implement a game clock. This indicates to me that your project has attempted to solve problems, in script, which are not suited to script. You have gotten help from Employees and Associates on numerous other threads, so I am confused as to why no performance issue threads have been created. The last post you made, on Jan 17th, was the thread regarding CPU usage near 50%, and made no mention of this issue. Somehow, in the 7 days between that post and this post, your team made a decision to abandon 6 months of work because you couldn't solve a problem that you turned off the solution for.
01/29/2007 (11:21 am)
You are making a game, and not a game engine. Which is why it is essential to understand and use the tools which are provided by the engine. I have worked with, and probably improved (at some point) every debugging tool provided by Torque. None of your posts relate to profiler problems, and since you have shown no hesitation to ask questions on other topics I have to question the decisions of your team regarding when to ask for help. You claim to have spent the time to fix the CPU problem, yet you turned off the profiler and didn't use it. This is a contradiction in terms. When you want to solve a performance issue, you use a profiler.Looking back at the forums, I see no reference to your problem except the thread about CPU usage near 50% at all times, which is perfectly normal for Torque. Last month you posted a thread here regarding problems using 'schedule' to implement a game clock. This indicates to me that your project has attempted to solve problems, in script, which are not suited to script. You have gotten help from Employees and Associates on numerous other threads, so I am confused as to why no performance issue threads have been created. The last post you made, on Jan 17th, was the thread regarding CPU usage near 50%, and made no mention of this issue. Somehow, in the 7 days between that post and this post, your team made a decision to abandon 6 months of work because you couldn't solve a problem that you turned off the solution for.
Torque Owner Anton Bursch
I agree 100%. And you know, the license for Torque or TGB is cheap, but your wasted time isn't.
This is a serious complaint and I hope nobody is dumb enough to treat it like it's some stupid 'Who needs shaders.' thread. This isn't an opinion thread. This is a major complaint by a company that wasted 6 months using TGB because of a major engine issue that if they had known of they wouldn't have used TGB to begin with.
And I don't mean any offense by this... but 6 months may not be a big deal to hobby developers, but to professional developers it's their living, and wasting 6 months of your companies time... it's insanity to think it's reasonable.
EDIT:
For the record: I am using TGB for a game that I'm working fulltime for and it's wonderful to use. I do however know that I have to account for these issues with TGB when I am using it. TGB has been making growth by leaps and bounds. In 18 months when this game is ready to ship... I expect that GG will have solved these engine issues. That is one reason why this thread is good. Because people should know about this and know what they have to work with. TGB is the best and only game engine in it's class, but it does need some serious work still. But damn it's fun to use.