Torque X huge memory leak - Fixed
by Giuseppe De Francesco · in Torque X 2D · 03/18/2010 (11:54 am) · 298 replies
Well,
I'm not sure this is the right place to post this, maybe the blog? At any rate I'm posting here now.
Torque X has a huge memory leak, but seriously! I was working to a RPG game for a start so I didn't really notice, but 2 weeks ago I started YASS, a multiplayer side scrolling space shooter: a lot of particles, really a lot, and a lot of projectiles as well.
To let you understand here follows a short video of the game:
This is the memory situation after 3 minutes of gameplay (single player!!!) the game slowed down a lot:
So, I searched the forum and I realized that this issue is an old one and GG never got a grip on it (go figure why...) so I decided to fix the problems myself because I really need to publish YASS ;)
Long story short: I revised the whole engine and made a lot of fixes, some of them really of great importance, so here it is the situation playing my game (as you can see after 9 minutes the game still is 20% less than the original 3 minutes!):
There are still many stuff to fix, but at any rate in the current state I'm able to publish the game ;) I will make the SVN diff files available upon request (free of charge: pino AT dftgames DOT com).
Cheers,
Pino
I'm not sure this is the right place to post this, maybe the blog? At any rate I'm posting here now.
Torque X has a huge memory leak, but seriously! I was working to a RPG game for a start so I didn't really notice, but 2 weeks ago I started YASS, a multiplayer side scrolling space shooter: a lot of particles, really a lot, and a lot of projectiles as well.
To let you understand here follows a short video of the game:
This is the memory situation after 3 minutes of gameplay (single player!!!) the game slowed down a lot:
So, I searched the forum and I realized that this issue is an old one and GG never got a grip on it (go figure why...) so I decided to fix the problems myself because I really need to publish YASS ;)
Long story short: I revised the whole engine and made a lot of fixes, some of them really of great importance, so here it is the situation playing my game (as you can see after 9 minutes the game still is 20% less than the original 3 minutes!):
There are still many stuff to fix, but at any rate in the current state I'm able to publish the game ;) I will make the SVN diff files available upon request (free of charge: pino AT dftgames DOT com).
Cheers,
Pino
About the author
In the software eng. field since 1981, in charge of R&D during last 10 years. IEEE Senior Member (and volunteer).
#142
04/05/2010 (6:44 pm)
Anyone else having a problem with minimizing? This one doesn't effect me as I'm only releasing on xbox but I've found objects stop rendering now after I minimize the game. Seams to be caused by one of the engine fixes. 
#143
@Duncan, the container full of sceneBinReferences is created with the SceneContainer, which is created by the scenegraph (or so I think). Look at the _CreateBins() method in TorqueCore/SceneGraph/SceneContainer.cs
I believe I have released the SceneContainerBinReference[] array when the SceneContainer is destroyed however I could be really wrong.
I am about to try and create my own pooling mechanism for my bullets, when I use the built in pool it will work pretty well for about 45 seconds and then literally grind to a halt. I still have something creating enough garbage that I get a 25-35 ms collection every 5-7 seconds. I know I have lots going on onscreen but still I would think it should be able to handle it.
I pooled most of my particles however, the particles that mount to something cannot be pooled as they then don't always stay mounted. Weirdness.
04/05/2010 (8:25 pm)
Matthew, I just gave it a try and no issues for me. I can minimize, and restore multiple times.@Duncan, the container full of sceneBinReferences is created with the SceneContainer, which is created by the scenegraph (or so I think). Look at the _CreateBins() method in TorqueCore/SceneGraph/SceneContainer.cs
I believe I have released the SceneContainerBinReference[] array when the SceneContainer is destroyed however I could be really wrong.
I am about to try and create my own pooling mechanism for my bullets, when I use the built in pool it will work pretty well for about 45 seconds and then literally grind to a halt. I still have something creating enough garbage that I get a 25-35 ms collection every 5-7 seconds. I know I have lots going on onscreen but still I would think it should be able to handle it.
I pooled most of my particles however, the particles that mount to something cannot be pooled as they then don't always stay mounted. Weirdness.
#144
@Matthew: I don't see any issue minimizing the game (beside the expected pause).
@Henry: I'll look into the pooling issue, it's strange.
04/06/2010 (1:50 am)
@Tony: still working on the issue but it's something that slips away each time I think to have understood the issue. Usually this means that it's a so evident problem that I don't see it :) Any help will be greatly appreciated!@Matthew: I don't see any issue minimizing the game (beside the expected pause).
@Henry: I'll look into the pooling issue, it's strange.
#145
FYI, for anyone using the TX3D engine I found an interesting issue. If you have multiple animation sequences in your DTS file and one that you are transitioning FROM doesn't have scaling, but the one you are transtioning TO does, one of the internal arrays doesn't get filled when the shape is loaded and it crashes.
My only solution was to have a scale from 0 to 0.00001 in all axis in the animation sequence that I really didn't want any in. That fixed it.
04/06/2010 (6:46 am)
@Pino: I don't know enough about the internals of the engine yet to be of much independant help. Is there anything particular that you would like me to try or play with that might help? Now that Zombie Sniper is in Playtest I will try to delve into the engine more.FYI, for anyone using the TX3D engine I found an interesting issue. If you have multiple animation sequences in your DTS file and one that you are transitioning FROM doesn't have scaling, but the one you are transtioning TO does, one of the internal arrays doesn't get filled when the shape is loaded and it crashes.
My only solution was to have a scale from 0 to 0.00001 in all axis in the animation sequence that I really didn't want any in. That fixed it.
#146
No worries though, if no one else is having the issue its nothing to worry about. :)
04/06/2010 (8:10 am)
The minimizing thing must be a video card issue. If the game minimizes due to break point everything still renders fine. the problem only turns up if I manually click the minimize button.No worries though, if no one else is having the issue its nothing to worry about. :)
#147
It doesn't sound like a specific graphics card issue. It will happen on any graphics card when your game loses the device context, if the game does not rebuild the resources that it stored on the graphics card. But, if you're only targeting xbox then no worries :)
04/06/2010 (8:26 am)
When you hit a break point the game doesn't minimize - it just switches to VS - at least that's what happens for me.It doesn't sound like a specific graphics card issue. It will happen on any graphics card when your game loses the device context, if the game does not rebuild the resources that it stored on the graphics card. But, if you're only targeting xbox then no worries :)
#148
Yesterday I coded some stuff for asynchronous loading of scenes (e.g. allows you to load a scene while displaying an animated 'loading' scene or any other scene you like). I'll be rolling it out to Pino for him to integrate it into the SVN project and then we can get you guys to give it a good thrashing to weed out any bugs. Seems solid at the moment, but needs real world testing with rich scenes.
It'll be a few days at least before it is available for you guys to test, but thought I'd let you know it's in the pipeline :)
04/06/2010 (8:33 am)
Just a quick heads up for you guys - a cool new feature is on it's way for those of you signed up to Pino's SVN project.Yesterday I coded some stuff for asynchronous loading of scenes (e.g. allows you to load a scene while displaying an animated 'loading' scene or any other scene you like). I'll be rolling it out to Pino for him to integrate it into the SVN project and then we can get you guys to give it a good thrashing to weed out any bugs. Seems solid at the moment, but needs real world testing with rich scenes.
It'll be a few days at least before it is available for you guys to test, but thought I'd let you know it's in the pipeline :)
#149
The issue with debugging it is once it happens I break, but then everything works as it should!
However I think there are a few little bugs in there and thats why we also have issues with pool with components and mounting pooled objects.
04/06/2010 (9:07 am)
What is really odd about the pooling, I have been tracing it out, it's like it gets overloaded and goes mad. the pooling code will automatically add the 4 core components, physics, collision, worldlimit and linkpoint. Sometimes even when you tell it not to. It will be running great and then suddenly I have 512MB of linkpoint components and value interfaces.The issue with debugging it is once it happens I break, but then everything works as it should!
However I think there are a few little bugs in there and thats why we also have issues with pool with components and mounting pooled objects.
#150
Henry I had an issue with mounted objects too. I have a post from about a year ago about it. http://www.torquepowered.com/community/forums/viewthread/92847.
A guy had a hack fix in that thread and said you could add SceneObject.SnapToMount() inside a ProcessTick() function.
To be honest my solution was just to stop using link points and wrote a component that just sets the objects position in relation to its parent. It sounds like that wouldn't work for you though as your creating so many objects. :(
04/06/2010 (9:48 am)
Duncan, again I must say you rock. I can't wait to try out async loading!Henry I had an issue with mounted objects too. I have a post from about a year ago about it. http://www.torquepowered.com/community/forums/viewthread/92847.
A guy had a hack fix in that thread and said you could add SceneObject.SnapToMount() inside a ProcessTick() function.
To be honest my solution was just to stop using link points and wrote a component that just sets the objects position in relation to its parent. It sounds like that wouldn't work for you though as your creating so many objects. :(
#151
@Duncan...async load sounds like it'll rock! Looking forward to it.
@Pino...thanks for continuing to let all of us reap the benefits of your findings!
--RB
04/06/2010 (10:53 am)
Been away a few days...have to get back up to date both in the forums and with the code updates. Hope all is well.@Duncan...async load sounds like it'll rock! Looking forward to it.
@Pino...thanks for continuing to let all of us reap the benefits of your findings!
--RB
#152
04/06/2010 (11:06 am)
hello, i was wondering if i could get the patch files as well, thanks for all your work on this. email: zero.division@yahoo.com
#153
For a few hours I won't get any emails because JustHost (where all my websites and emails are) made a mess with all my stuff do I'm migrating all my personal stuff to HostGator now... quite a busy time :(
A word of advice: if you need a hosting provider... stay away from JustHost.
04/06/2010 (2:38 pm)
Hi Alex, sure you can, just, I'll send you the invitation to the SVN repository.For a few hours I won't get any emails because JustHost (where all my websites and emails are) made a mess with all my stuff do I'm migrating all my personal stuff to HostGator now... quite a busy time :(
A word of advice: if you need a hosting provider... stay away from JustHost.
#154
I am trying to figure it out, hopefully I can get somewhere with it. I know I am not at some of your levels when it comes to this kind of thing.
04/06/2010 (6:36 pm)
T2DTileObject doesn't dispose or it is another part of the Scene changing that it doesn't get released. Anyway I kept getting a growing growing memory issue I have 4 levels with tiles, 40*40. I am trying to figure it out, hopefully I can get somewhere with it. I know I am not at some of your levels when it comes to this kind of thing.
#155
04/06/2010 (10:02 pm)
Seriously...you guys rock. This will be a huge help. The release was already on it's way to QA and I'm not sure if these changes made it into the build. If not, I'll get another release soon after.
#156
It's good to know that these improvements that we're relying on to get out games up to production-quality will be there for us in subsequent builds.
Can't say it enough...my hat's off to Pino for an amazing bit of optimization.
--RB
04/07/2010 (4:15 am)
@Eric...thanks for integrating Pino's changes (and everyone else who's contributed to the online repos).It's good to know that these improvements that we're relying on to get out games up to production-quality will be there for us in subsequent builds.
Can't say it enough...my hat's off to Pino for an amazing bit of optimization.
--RB
#157
04/07/2010 (6:38 am)
@Eric: I assume you are talking about a GG release. If that is the case don't release any of the 3D fixes yet. There is still a major problem that Pino and I are trying to figure out in the 3D stuff.
#158
He might be able to help Pino and Tony to iron out the 3D problem.
Might be worth reaching out to him...especially if the expectation is that these fixes will become part of the TX code base.
--RB
04/07/2010 (6:46 am)
@Eric...It'd be nice to get John K's involvement in this thread at some point.He might be able to help Pino and Tony to iron out the 3D problem.
Might be worth reaching out to him...especially if the expectation is that these fixes will become part of the TX code base.
--RB
#159
Maybe they used the sources that I sent as a starting point to go ahead on their own? We'll see... I keep my fingers crossed :)
04/07/2010 (7:09 am)
Well I don't think GG will include the fixes from my repository because the last patches they got are the one sent by email, quite outdated... their repository invitations are still pending. Maybe they used the sources that I sent as a starting point to go ahead on their own? We'll see... I keep my fingers crossed :)
#160
* when the game is producing and disposing objects very quickly the renderer processes an rendering event while the object is being removed so getting a Code 4 because of an Assert.
* Starting the game while opening and closing the Guide (on the Xbox) multiple times produces a Code 4.
I'm submitting the fix now.
04/07/2010 (6:29 pm)
I 've just found two more issues with the GUI renderer: * when the game is producing and disposing objects very quickly the renderer processes an rendering event while the object is being removed so getting a Code 4 because of an Assert.
* Starting the game while opening and closing the Guide (on the Xbox) multiple times produces a Code 4.
I'm submitting the fix now.
Torque Owner Tony Pitman
Default Studio Name
I have a 3D game and was trying to integrate your changes to the core and 3D stuff into it. When I added your changes the framerate went to pot. You did not see the results in the FPS sample, but then later told me you were seeing the same thing in the Physics sample.
I was wondering if you had gotten anywhere on that? I know you are working on a bunch of stuff at once, so I am just asking to see if there is an update.