Memory Leak in particle engine
by Jon Fernback · in Torque Game Engine · 01/06/2006 (11:32 am) · 21 replies
Hi,
I was just wondering if anyone else has noticed that the particles in TGE are not being cleaned up properly? We just made this discovery earlier this week on a project we are working on. At first we were not sure if the leak was caused by something we were doing or the stock engine, but after running some tests today I discovered that the demo fps has the same problem.
First, edit crossbow.cs so the rate of fire on the xbow is really fast and comment out the line that modifies the players inventory (that way you don't run out of ammo).
Then start up a mission and check out how much ram the game is taking. For us it is around 83 megs. Now fire off bolts as fast as you can for about 10 minutes, the ram usage will climb considerably and never drop back down. (For me it jumped up to near 100 megs)
Now if you leave the mission and restart, the base memory footprint will grow to be close to whatever it was when you left, not the original 83megs.
We noticed this in our new prototype when the memory footprint just kept growing the longer we played, and decided to hook up the memory manager resource. Sure enough many allocations from the particles and particle engine were not cleaned up after their lifetime ended or even after the mission ended. (Which would be a huge problem for games that lasted hours per mission or were trying to run on low memory footprints)
Here is what we figured out so far:
The problem seems to be in particleEngine.cc
void ParticleEmitter::processTick(const Move*)
if (mDeleteOnTick == true)
deleteObject();
mDeleteOnTick is being set correctly in the advanceTime function, but processTick isn't being called for every emitter after advanceTime is called. We think maybe it's an issue w/ ProcessList::advanceObjects() either a) not containing the emitter in it's list, or b) mProecessTicks not being true.
If anyone else has any info on this problem we're looking to fix that would be great. We'll keep you posted on any progress.
I was just wondering if anyone else has noticed that the particles in TGE are not being cleaned up properly? We just made this discovery earlier this week on a project we are working on. At first we were not sure if the leak was caused by something we were doing or the stock engine, but after running some tests today I discovered that the demo fps has the same problem.
First, edit crossbow.cs so the rate of fire on the xbow is really fast and comment out the line that modifies the players inventory (that way you don't run out of ammo).
Then start up a mission and check out how much ram the game is taking. For us it is around 83 megs. Now fire off bolts as fast as you can for about 10 minutes, the ram usage will climb considerably and never drop back down. (For me it jumped up to near 100 megs)
Now if you leave the mission and restart, the base memory footprint will grow to be close to whatever it was when you left, not the original 83megs.
We noticed this in our new prototype when the memory footprint just kept growing the longer we played, and decided to hook up the memory manager resource. Sure enough many allocations from the particles and particle engine were not cleaned up after their lifetime ended or even after the mission ended. (Which would be a huge problem for games that lasted hours per mission or were trying to run on low memory footprints)
Here is what we figured out so far:
The problem seems to be in particleEngine.cc
void ParticleEmitter::processTick(const Move*)
if (mDeleteOnTick == true)
deleteObject();
mDeleteOnTick is being set correctly in the advanceTime function, but processTick isn't being called for every emitter after advanceTime is called. We think maybe it's an issue w/ ProcessList::advanceObjects() either a) not containing the emitter in it's list, or b) mProecessTicks not being true.
If anyone else has any info on this problem we're looking to fix that would be great. We'll keep you posted on any progress.
About the author
#2
I'm thinking that this could of been the cause of the 400 mb mem leak we had in our game. Perhaps you could just delete the particles when the emitter is removed as a fall back(haven't looked at the code yet)?
01/06/2006 (12:30 pm)
Wow!I'm thinking that this could of been the cause of the 400 mb mem leak we had in our game. Perhaps you could just delete the particles when the emitter is removed as a fall back(haven't looked at the code yet)?
#3
I put some logging in the onAdd, and the onRemove methods in the ParticleEmitter class. An interesting thing I've noticed is that onRemove is being called far more times than OnAdd. Just to clear things up, in the onAdd method, I'm logging just before the function returns true, so the Emitter is definitely(AFAIK) added.
I also put some logging in the ParticleEmitter's constructor and destructor, and it looks like the constructor is being called several more times than the destructor. If I'm understanding things correctly, shouldn't the constructor and destructor be called the same number of times throughout the game session? Perhaps someone who's more familiar with the engine could answer that one...
01/06/2006 (1:51 pm)
From the looks of things, it looks like ParticleEmitter::OnRemove is already(should be anyway) doing just that.I put some logging in the onAdd, and the onRemove methods in the ParticleEmitter class. An interesting thing I've noticed is that onRemove is being called far more times than OnAdd. Just to clear things up, in the onAdd method, I'm logging just before the function returns true, so the Emitter is definitely(AFAIK) added.
I also put some logging in the ParticleEmitter's constructor and destructor, and it looks like the constructor is being called several more times than the destructor. If I'm understanding things correctly, shouldn't the constructor and destructor be called the same number of times throughout the game session? Perhaps someone who's more familiar with the engine could answer that one...
#4
01/08/2006 (1:07 am)
Issue #1082. Thanks for the research on this one, guys!
#5
What I found is that the secondary emitters of an explosion weren't being cleaned up. (Emitter[0], Emitter[1], etc).
I changed the code on line 1108 in explosion.cc from this:
to this:
Once that was changed, it seemed to take care of the leak showing up in explosion.cc as far as DumpUnflaggedAllocs was concerned.
At first, I thought that the reason it wasn't being cleaned up was because deleteWhenEmpty() wasn't being called. For whatever reason, that wasn't the case. A call to emitParticles from the emitter being created also seems neccessary, because without it, the emitter doesn't get cleaned up. This also changed how the secondary effects are being rendered, but I'm assuming it's because I'm using the explosion datablocks particledensity and particle radius values. There might be some more modifications needed to the explosiondatablock to set those values for the additional emitters.
BUT... When viewing the memory usage in windows task manager, it's still showing RAM usage increasing as the game is running. My next step is going to set some logging in the resource manager to see if resources are being loaded redundantly throughout the game session.
If anyone else has any ideas, they'd be greatly appreciated!
Thanks!
EDIT: On further review, it looks like this code in emitParticles may be the reason emitParticles has to be called:
I'm assuming that since the cleanup for particle emitters is being done in the processTicks method of the particleEmitter, and processTicks is called for objects in gClientProcessList (check out gameProcess.cc), then the particle won't be added to gClientProcessList unless emitParticles is called after the emitter's creation. I could be wrong though.
01/09/2006 (8:02 am)
Ok, I made a little progress... What I did was went to the spots in code where DumpUnflaggedAllocs was showing as a memory leak after exiting a mission, and added logging to log the ID of the object being created with a "new". (After registerObject was called of course). I also added logging where the particles were being deleted to log the ID's of emitters being deleted.What I found is that the secondary emitters of an explosion weren't being cleaned up. (Emitter[0], Emitter[1], etc).
I changed the code on line 1108 in explosion.cc from this:
for( int i=0; i<ExplosionData::EC_NUM_EMITTERS; i++ )
{
if( mDataBlock->emitterList[i] != NULL )
{
ParticleEmitter * pEmitter = new ParticleEmitter;
pEmitter->setDataBlock( mDataBlock->emitterList[i] );
if( !pEmitter->registerObject() )
{
Con::warnf( ConsoleLogEntry::General, "Could not register emitter for particle of class: %s", mDataBlock->getName() );
SAFE_DELETE(pEmitter);
}
mEmitterList[i] = pEmitter;
}
}to this:
for( int i=0; i<ExplosionData::EC_NUM_EMITTERS; i++ )
{
if( mDataBlock->emitterList[i] != NULL )
{
ParticleEmitter * pEmitter = new ParticleEmitter;
pEmitter->setDataBlock( mDataBlock->emitterList[i] );
if( !pEmitter->registerObject() )
{
Con::warnf( ConsoleLogEntry::General, "Could not register emitter for particle of class: %s", mDataBlock->getName() );
SAFE_DELETE(pEmitter);
}
else
{
pEmitter->emitParticles(getPosition(), mInitialNormal, mDataBlock->particleRadius,
Point3F(0, 0, 0), U32(mDataBlock->particleDensity * mFade));
pEmitter->deleteWhenEmpty();
}
mEmitterList[i] = pEmitter;
}
}Once that was changed, it seemed to take care of the leak showing up in explosion.cc as far as DumpUnflaggedAllocs was concerned.
At first, I thought that the reason it wasn't being cleaned up was because deleteWhenEmpty() wasn't being called. For whatever reason, that wasn't the case. A call to emitParticles from the emitter being created also seems neccessary, because without it, the emitter doesn't get cleaned up. This also changed how the secondary effects are being rendered, but I'm assuming it's because I'm using the explosion datablocks particledensity and particle radius values. There might be some more modifications needed to the explosiondatablock to set those values for the additional emitters.
BUT... When viewing the memory usage in windows task manager, it's still showing RAM usage increasing as the game is running. My next step is going to set some logging in the resource manager to see if resources are being loaded redundantly throughout the game session.
If anyone else has any ideas, they'd be greatly appreciated!
Thanks!
EDIT: On further review, it looks like this code in emitParticles may be the reason emitParticles has to be called:
if (mParticleListHead != NULL && mSceneManager == NULL)
{
gClientSceneGraph->addObjectToScene(this);
gClientContainer.addObject(this);
gClientProcessList.addObject(this);
}I'm assuming that since the cleanup for particle emitters is being done in the processTicks method of the particleEmitter, and processTicks is called for objects in gClientProcessList (check out gameProcess.cc), then the particle won't be added to gClientProcessList unless emitParticles is called after the emitter's creation. I could be wrong though.
#6
Before you go adding lots of logging code, do note that there's already functions to dump loaded resources & the like. :)
Anyway, I'd STRONGLY recommend using the memory logging, any growth after load is going to show up there much more clearly than in the resource manager! (Since anything the resmanager is doing will show up in the memory manager, too.)
01/09/2006 (11:19 am)
Hey Jacob,Before you go adding lots of logging code, do note that there's already functions to dump loaded resources & the like. :)
Anyway, I'd STRONGLY recommend using the memory logging, any growth after load is going to show up there much more clearly than in the resource manager! (Since anything the resmanager is doing will show up in the memory manager, too.)
#7
I did add some logging to it, and the problem's not w/ the resource manager. Actually, I'm not sold that it's anything to do with the engine at all at this point, especially now that the unflagged memory issues I had posted above have been taken care of.
I played through a game session without any audio, and Torque didn't seem to be eating ram like it did when there were sounds playing, so at this point, I'm thinking it's something to do with the OpenAL implementation.
Thanks again!
01/09/2006 (2:12 pm)
Nice, thanks Ben!I did add some logging to it, and the problem's not w/ the resource manager. Actually, I'm not sold that it's anything to do with the engine at all at this point, especially now that the unflagged memory issues I had posted above have been taken care of.
I played through a game session without any audio, and Torque didn't seem to be eating ram like it did when there were sounds playing, so at this point, I'm thinking it's something to do with the OpenAL implementation.
Thanks again!
#8
01/09/2006 (3:04 pm)
Hmm, that's pretty scary. One reason we're working hard on the audio rewrite. :)
#10
Do *not* use AlxGetWavLen() if your concerned about memory use. The function leaks memory for breakfest.
01/11/2006 (6:07 am)
Just a note about the audio.Do *not* use AlxGetWavLen() if your concerned about memory use. The function leaks memory for breakfest.
#11
01/11/2006 (11:29 am)
It'll be out everywhere we can put it.
#12
Unless alGetBufferi leaks memory, which would then mean the problem is in openAL itself right??
Unless I'm missing something?
01/11/2006 (1:52 pm)
Looking at AlxGetWaveLen(), are you sure thats the function you mean? All it does is make 4 calls thru the openAL api to alGetBufferi() to get back 4 int values which are local in scope and then do a calculation based on those values.Unless alGetBufferi leaks memory, which would then mean the problem is in openAL itself right??
Unless I'm missing something?
#13
Its kind of a long train of functions...
emitParticles() is set to be called by the explosion code in updateEmitters()
updateEmitters() is called in advanceTime().
advanceTime() is called for all members of the clientProcessList.
Now, the explosions were never being added to the clientProcessList in TGE 1.4. An object created on a client by the client considers it to be the server object. So the default onAdd() behavior is to add it to the server list. Client list is for ghosted objects which explosions are not.
You'll notice at line 736 in explosions.cc that there is an if block that adds it to the client process list. Either a break point in there or a Con::printf echo will show that the code block is never executed. If its not a serverObject, it must be a clientObject (which is what its checking), but as explained above, explosions are created as serverObjects initially, so the conditions cant be met, and its never part of the client process list.
Which leads back to the emitters not being called.
This was actually investigated to get the explosionShape field to work again in 1.4. Removing that particular if (!isServerObject) { } fixed it for me, and made the updateEmitters get called.
I actually went and changed all the !isServerObject() to a GameConnection check to be consistent with the start of the onAdd() function.
Adding Jacobs change afterwards actually introduced a crash for my fixed version.
01/17/2006 (11:31 am)
I actually cleaned up the problem for the explosions emitters entirely by chance last night.Its kind of a long train of functions...
emitParticles() is set to be called by the explosion code in updateEmitters()
updateEmitters() is called in advanceTime().
advanceTime() is called for all members of the clientProcessList.
Now, the explosions were never being added to the clientProcessList in TGE 1.4. An object created on a client by the client considers it to be the server object. So the default onAdd() behavior is to add it to the server list. Client list is for ghosted objects which explosions are not.
You'll notice at line 736 in explosions.cc that there is an if block that adds it to the client process list. Either a break point in there or a Con::printf echo will show that the code block is never executed. If its not a serverObject, it must be a clientObject (which is what its checking), but as explained above, explosions are created as serverObjects initially, so the conditions cant be met, and its never part of the client process list.
Which leads back to the emitters not being called.
This was actually investigated to get the explosionShape field to work again in 1.4. Removing that particular if (!isServerObject) { } fixed it for me, and made the updateEmitters get called.
I actually went and changed all the !isServerObject() to a GameConnection check to be consistent with the start of the onAdd() function.
Adding Jacobs change afterwards actually introduced a crash for my fixed version.
#14
Sebastien, thanks for posting this with an explanation. I don't know how long it would have taken me to get my brain around the fact that an object created on the client that isn't a ghost is assumed to be a server object.
Jacob, I think the problem with your change is that, once Sebastien's fix is implemented, you would also need to take out the call to UpdateEmitters() in advanceTime(). All UpdateEmitters() does is call emitParticles() on each of the secondary emitters. However, the call to deleteWhenEmpty() means the emitters may have been deleted since the last advanceTime(). You would also need to remove the calls to deleteWhenEmpty() in onRemove().
Please note, I may have misinterpreted the code!
I'm still the new kid on the block when it comes to this code base.
edit: A question I have is, which way is correct? The existing code continually calls emitParticles() during advanceTime() while Jacob's way would just call emitParticles() on the emitters once.
[Heading back to do some more testing :)]
Thanks
Todd
01/18/2006 (12:23 pm)
Hi Guys,Sebastien, thanks for posting this with an explanation. I don't know how long it would have taken me to get my brain around the fact that an object created on the client that isn't a ghost is assumed to be a server object.
Jacob, I think the problem with your change is that, once Sebastien's fix is implemented, you would also need to take out the call to UpdateEmitters() in advanceTime(). All UpdateEmitters() does is call emitParticles() on each of the secondary emitters. However, the call to deleteWhenEmpty() means the emitters may have been deleted since the last advanceTime(). You would also need to remove the calls to deleteWhenEmpty() in onRemove().
Please note, I may have misinterpreted the code!
I'm still the new kid on the block when it comes to this code base.
edit: A question I have is, which way is correct? The existing code continually calls emitParticles() during advanceTime() while Jacob's way would just call emitParticles() on the emitters once.
[Heading back to do some more testing :)]
Thanks
Todd
#15
There is a deleteWhenEmpty in the onRemove for those subEmitters which handles that call. Which would have always been called before, but must not work without the emitParticle call. Which should likely be fixed in some way.
01/18/2006 (12:52 pm)
I believe the multiple calls to emitParticles is so that the particles *move* with the explosion as the only real updated part is that it sets its position to the current explosion position.There is a deleteWhenEmpty in the onRemove for those subEmitters which handles that call. Which would have always been called before, but must not work without the emitParticle call. Which should likely be fixed in some way.
#17
William, both Sebastian's and my changes do effectively the same thing, which is ensure the emitter is added to the client list. His way is actually the cleaner way to do it, and the way I'd recommend doing it, as it takes care of the issue in the emitter code, rather than in the explosion code.
The point of calling update emitParticles after 'new' ing an emitter wasn't to actually emit the particles on that line of code, it was to get the emitter in the clientList so that the emitter would actually get processed. (IE calling advanceTime on the emitter in the client process loop so emit particles would be continuously be called, see ProcessList::advanceObjects in gameprocess.cc). It was essentially a hack to get the emitter in the client loop :)
What was happening with the memory leak before, was that the engine was creating new particle emitters, and registering them, but never adding them to the client list, which kept them from being processed. Since all of the cleanup code for emitters is in the processTick method, the emitters weren't being cleaned up, as well as not rendering.
01/18/2006 (5:35 pm)
Quote:
The existing code continually calls emitParticles() during advanceTime() while Jacob's way would just call emitParticles() on the emitters once.
William, both Sebastian's and my changes do effectively the same thing, which is ensure the emitter is added to the client list. His way is actually the cleaner way to do it, and the way I'd recommend doing it, as it takes care of the issue in the emitter code, rather than in the explosion code.
The point of calling update emitParticles after 'new' ing an emitter wasn't to actually emit the particles on that line of code, it was to get the emitter in the clientList so that the emitter would actually get processed. (IE calling advanceTime on the emitter in the client process loop so emit particles would be continuously be called, see ProcessList::advanceObjects in gameprocess.cc). It was essentially a hack to get the emitter in the client loop :)
What was happening with the memory leak before, was that the engine was creating new particle emitters, and registering them, but never adding them to the client list, which kept them from being processed. Since all of the cleanup code for emitters is in the processTick method, the emitters weren't being cleaned up, as well as not rendering.
#18
Ok, I did not fully grasp what you were doing.
Thank you for taking the time to explain it to me. It is really helpful to me as I try to get my arms around the engine.
Thanks again.
Todd
01/18/2006 (6:39 pm)
Ahhh,Ok, I did not fully grasp what you were doing.
Thank you for taking the time to explain it to me. It is really helpful to me as I try to get my arms around the engine.
Thanks again.
Todd
#19
I am also facing the problem of "Progressive slowdown problem" as you move in the mission. We have used lots of particle emitters and on removing emitters from mission the application seems to behave fine.
I am not using explosion.cc. I could not see any fix suggested for particle emitters problem? Is that issue solved?(which seems to be related with calling of deleteWhenEmpty() methos as being discussed in forums)
Any info / guidance on it will be helpful. I am using TGE 1.3
03/26/2006 (11:43 pm)
Hi all,I am also facing the problem of "Progressive slowdown problem" as you move in the mission. We have used lots of particle emitters and on removing emitters from mission the application seems to behave fine.
I am not using explosion.cc. I could not see any fix suggested for particle emitters problem? Is that issue solved?(which seems to be related with calling of deleteWhenEmpty() methos as being discussed in forums)
Any info / guidance on it will be helpful. I am using TGE 1.3
#20
http://www.garagegames.com/index.php?sec=mg&mod=resource&page=view&qid=8156
and it is running throughtout the game for a filler effect, but when i try to switch levels it makes a crash and i do not know why. You guys got any suggestions on this issue??
08/30/2007 (12:38 pm)
I am also having a problem with the particle emitter where it crashes my game when i try to switch levels. I have a mounted particle emitter on my guy using the code from chris lambard at http://www.garagegames.com/index.php?sec=mg&mod=resource&page=view&qid=8156
and it is running throughtout the game for a filler effect, but when i try to switch levels it makes a crash and i do not know why. You guys got any suggestions on this issue??
Torque Owner Jake Oster
FlagCurrentAllocs();
then run the mission for around minute, rapidly firing projectiles w/ explosion emitters, exit the mission, and DumpUnflaggedAllocs(), I see hundreds (if not a thousand) references to lines of code that are creating new emitters. The two most prominent are
bem->emitter = new ParticleEmitter; in shapeBase.cc
and
ParticleEmitter * pEmitter = new ParticleEmitter; in explosion.cc
I've added the code Scott Richards provides in this thread
http://www.garagegames.com/mg/forums/result.thread.php?qt=21534
But that doesn't seem to be the issue.
I did notice the comments in ProcessList::AdvanceObjects about objects being deleted in their process method, which appears to be how the particle engine does it, and at this point am wondering if that could somehow be related to the issue...
I guess at this point I'm at a loss, and will keep digging; any advice or fixes would be greatly appreciated!
Thanks!