Torque X huge memory leak - Fixed

#121 I've just committed a few more improvements.
Please be sure to accept all the invitations!

* The TX2D licensees get 2 invitation, one for the shared Core and one for the 2D library.
* The TX3D licensees get 2 invitation, one for the shared Core and one for the 3D library.
* The TX2D and TX3D licensees get 3 invitation, one for the shared Core, one for the 2D and one for the 3D library.

#122 Just comitted new improvements.

#123 Hi Giuseppe, please add me to the list: diegoleao@gameblox.net

It has been a while since I posted here, and I'm immensely happy that this community is making progress. You're being a big part of that, so thank you very much!

#124 Hey,

I've been out of the country for two months and taking a break. Seeing a post with 120+ responses seemed crazy. I've been reading over all the work you guys have been putting in to this problem and would like to add a sincere thanks for it all.

I'm new to TX and game development. I would have never been able to find any of this and by the looks of the GG staff this may have never gotten fixed.

I'm going to get all the programs you talked about on the first page, could I get an invite to the list?

Thanks again for all your guys' work.

#125 Pino did you see this: www.torquepowered.com/community/blogs/view/19533/2#comment-154709

BTW my real work should be done tomorrow and I can start messing with this, at least over the weekend.

#126 Hi Henry,

just seen it: it's nice that we got some acknowledgement about the work done ;)

#127 Hi guys,

I’ve found another big issue which stopped me from publishing the game, again about particles, but it can be more wide so I’d like to hear from you about what else could have the same problem. Here’s the issue:

In YASS I’ve planned to have comets traversing the screen. Comets are made by particles (mostly for the tail) mounted on a static sprite. I have a few different templates for them, so I started cloning them, then generating a random motion and assigning to the tail the proper angle (_particleEffect.CurrentEffectData.EmitterDataList[0].FixedForceAngle) computed from the velocity, and… here comes the problem: when changing that angle in one clone then all of them change right away!

This happens because the objects are not really cloned by the Clone() method: they are all passed on by reference. Now… this can make sense in general because a lot of resources can be shared without any collateral effect, but it’s not the case for the particles and maybe there are other situations that I can’t imagine now where the same happens with other objects, that’s why I’m asking all of you to think about this and let me know if there is any other use case that might present a similar issue.

BTW, I’ve also exposed as Public (read-only) the SceneObject’s MountedObjects so it’s now possible to look into that from an external game manager ;)

I look forward to getting your suggestions about other unwanted referenced copy.

#128 I can't think of any issues I've had, but I'll keep an eye out. Its good to know that about particles though.

I can't wait to put in the new updates tonight :) Didn't get a chance last night :).

#129 Sounds like the particles needs a flag that can be set to clone shallow or deep (defaults to shallow).

#130 Duncan,

the TorqueCloneDeep flag (or it's absence, meaning shallow) has no effect whatsoever on the cloning process: it's only used to perform an integrity check if TRACE is defined. The actual "cloning"`is executed by calls at the CopyTo method, where the objects are just assigned by reference (obj2.xxx = this.xxx) thus I had to create a new ParticleEmitterData object and populate it.

#131 Yep, that's what I meant - adding something to handle the shallow/deep copying depending on a flag. I didn't know about the TorqueCloneDeep flag :-)

#132 <removed>

#133 I have the _scenecontainerbinreterence[] releasing now. In T2DSceneGraph.cs you need to ad an OnUloaded method at line 325:

public override void OnUnloaded()
{
     base.OnUnloaded();
     this._container.Dispose();
}

then in SceneContainer.cs line 927:

for (uint i = 0; i < _sceneBinArray.Length; i++)
{
    if (_sceneBinArray[i] != null)
        _sceneBinArray[i].Reset(); // Just to make sure to not create a circular reference of disposed objects
    _sceneBinArray[i] = null;
}

It seems that something else is still slowly building up.

#134 Awesome! Thanks Henry.

Some things I've been working on.

I've managed to Drastically reduce my issues/memory leaks!

I changed the way my loading code works. I moved some objects around in my scene files. I now never unload my shared data scene file. As a result I save about 84,000 k in memory that I was loosing every time I unloaded (as my components and animation data would hang around)

It seams that unloading two scene files and then reloading them can cause some issues, that leaving one scene in memory is solving for me. I was having null reference errors and such that Pino was working with me to fix but these are all gone now that I stopped unloading both scene files.


Also here is a fix to a huge leak in the GUI. Though It doesn't show up as a leak in the profiler. If you use a GUI it will continually create RenderInstances until it reaches 168,131 instances and then stops, sucking up about 30 megs of memory. These instances are never removed.

To fix it in the Util.cs DrawUtil class go to the BitmapStretchSR method and add this line at the very end after "material.CleanupEffect();"

SceneRenderer.RenderManager.FreeInstance(_workingRenderInstance);

This should lower your memory footprint by up to 30 megs :).

The fix is from this old post from a year ago that I dug back up :). http://www.torquepowered.com/community/forums/viewthread/92771

#135 @Henry: when I went back to look in more detail I didn't actually find that _scenecontainerbinreterence[] was leaking. What I found was that Windows' more advanced garbage collector was simply taking it's sweet time collecting those references - which can make it look like a leak.

What nulling out the references does is to reduce the amount of work the garbage collector has to do to collect those objects, since there is a less complex chain of references to navigate when determining what needs gc'ing. Indeed, you would expect the _scenecontainerbinreterence[] will get collected eventually, since it is only pinned by the container, which is itself only pinned by the scenegraph - null references just speeds this up (and gc also takes less time, so less likely to glitch your framerate).

On the other hand I may have missed something in my reading of the code. Or maybe the way I am using TX is just not causing the leak to occur. If you have a small project example that leaks _scenecontainerbinreterence[] then I would be interested to look at it and see what is going on.

#136 @Duncan It seems that every time I unload a level it leaves a SceneContainerBinReference array of 256k in memory. even calling GC.Collect() would leave these structures. In my game by the fourth level I would have an extra meg of memory just eaten up and it would start to chug.

#137 Hi,

I'm tracking this issue but the option to call the _container.Dispose() directly on the OUnload delegate makes crash 2 out of 5 of my games. It's all about timing I guess.

Tracking in real time the memory allocation I see that the BinReferences allocation is proportional to the GC Generation #0 allocation so that doesen't look like a memory leak but is the usual GC issue (and it'll be no use the GC.Collect because it'll do whatever it wants anyway).

Over time all those instances are collected with only one exception: the GUI objects tied by GFXDynamicVertexBufferPCTTBN are held by an EventHandler root reference. I'm looking into that right now. These GFXDynamicVertexBufferPCTTBN are weakly rooted so they will be disposed eventually, but they keep too long to let go so something is not right.

#138 Absolutely - code that cleans up after itself is good practice even when there is a garbage collector involved. Looking around in the code this is something that TX often fails to do though and I think a lot of the framerate glitching people are experiencing is down to that. C# can make us lazy in that regard and I don't think the issue is promoted as much as it should be in XNA game dev - many new developers aren't even aware of the need to help out the garbage collector.

The engine mods in this thread largely address that issue (rather than memory leaks per se) and already people can see a big difference in their games (indeed, as a rule memory leaks don't cause framerate glitching unless the thing that is leaking is also directly or indirectly involved in the game processing and not simply just hanging around taking up space).

#139 Causes a crash? That is what I was afraid of, however in my code I load a GUI then unload the scene. I wonder if it has to do with you not using the GUI? That the scene is still in use when you unload?

#140 @Henry & Pino: Also bear in mind that scene objects don't always get associated with the scenegraph that they should be (unless you take steps to ensure it - see earlier post). Possibly that might cause an issue in this instance as it is the scenegraph that is disposing the container?