Object Pooling in .NET??
by Paul Modzelewski · in Torque X 2D · 03/09/2007 (11:02 pm) · 11 replies
To begin with, the ObjectPooler class is stylistically problematic, using FindPooledObject and RecycleObject to manage memory is a lot like using new and delete, with most of the same drawbacks. It's also technically problematic; ObjectPooler is a memory manager that sits on top of the CLR's memory manager, and stacking can cause all kinds of problems (like plugging a Surge Protector into a UPS).
Have you guys considered what will happen to pooled objects from the perspective the CLR's generational garbage collector? How much testing/profiling has this code received?
The TorqueX overview doc says this:
"...our guess(??) is that you will want to turn on object pooling for projectiles and explosions and other things which are created many and often, but probably not turn it on for game objects with longer lifetimes."
The .Net Developers Guide on MSDN says this:
"For managed short-term objects that are created and then go out of scope between collections, the cost of allocating and releasing memory is extremely low. In the .NET Framework, the garbage collector is actually optimized to manage objects with short lifetimes."
Using a pool to bypass Garbage Collection goes against the grain of managed programming. Without pooling, the objects live a short life and are quickly collected from the Gen 0 heap when memory is needed. With pooling, the objects get promoted to older generations, where they are less likely to be collected and when they are eventually collected it will take longer (not to mention the overhead of caching/retrieving objects to/from the pool).
On top of that, putting short-lived objects in the pool means the GC will have to collect objects from older generations to free memory, which takes longer. Finally, the GC still has to iterate over all the objects in the pool when looking for unreferenced objects to free, even though they will never be unreferenced.
There are some implementation problems with the class as well. The only way to empty the pool is to manually get every object, meaning you have to remember what types of objects are in the pool. The methods are all static (why isn't the whole class static?), so the ObjectPooler can't be deleted. Objects placed into the pooler could easily be stuck there for the life of the application as well. Shouldn't there be some sort of "Clear" method so that pooled objects can be released?
Even if there are a few situations where "proper" use of the ObjectPooler might result in better performance, there are an unlimited number of ways to screw things up, and this class will give .Net novices more than enough rope to hang themselves. Unless I'm missing something, I'd recommend pulling it from the library.
Have you guys considered what will happen to pooled objects from the perspective the CLR's generational garbage collector? How much testing/profiling has this code received?
The TorqueX overview doc says this:
"...our guess(??) is that you will want to turn on object pooling for projectiles and explosions and other things which are created many and often, but probably not turn it on for game objects with longer lifetimes."
The .Net Developers Guide on MSDN says this:
"For managed short-term objects that are created and then go out of scope between collections, the cost of allocating and releasing memory is extremely low. In the .NET Framework, the garbage collector is actually optimized to manage objects with short lifetimes."
Using a pool to bypass Garbage Collection goes against the grain of managed programming. Without pooling, the objects live a short life and are quickly collected from the Gen 0 heap when memory is needed. With pooling, the objects get promoted to older generations, where they are less likely to be collected and when they are eventually collected it will take longer (not to mention the overhead of caching/retrieving objects to/from the pool).
On top of that, putting short-lived objects in the pool means the GC will have to collect objects from older generations to free memory, which takes longer. Finally, the GC still has to iterate over all the objects in the pool when looking for unreferenced objects to free, even though they will never be unreferenced.
There are some implementation problems with the class as well. The only way to empty the pool is to manually get every object, meaning you have to remember what types of objects are in the pool. The methods are all static (why isn't the whole class static?), so the ObjectPooler can't be deleted. Objects placed into the pooler could easily be stuck there for the life of the application as well. Shouldn't there be some sort of "Clear" method so that pooled objects can be released?
Even if there are a few situations where "proper" use of the ObjectPooler might result in better performance, there are an unlimited number of ways to screw things up, and this class will give .Net novices more than enough rope to hang themselves. Unless I'm missing something, I'd recommend pulling it from the library.
About the author
#2
I created a class that implements ObjectPooler.IResetable. The class copies data from a static MemoryStream into its own collection to simulate actual work and use a little CPU and memory. In Game.Update I create some of these objects and have them process their data. In pooling mode, 80% of these object are randomly recycled into the pool.
The app toggles pooling once per minute and logs the average and max frame times for the last 60 seconds. Here's a piece of the log file. Avg and Max are frame timings in milliseconds, and the best and worst individual frame times are excluded from the average.
I also used perfmon (the GC counters are in the ".Net CLR Memory" section) and took some screen caps of its output. In both images, yellow is the number of bytes promoted from Gen 0 to Gen 1, blue is the number of bytes promoted from Gen 1 to Gen 2, skinny red is the Gen 1 heap in bytes, and thick red is the Gen 2 heap in bytes (note, the Gen 2 heap is drawn at a smaller scale, it's much larger than the Gen 1 heap).
The first image shows one minute of data with pooling turned on. Notice how much data is getting promoted to Gen 1 and Gen 2. At the end, promotions from Gen 1 increase the Gen 2 heap to hundreds of MB, which is then collected. That mammoth collection is the cause of the 700 ms Max frame time.
The second image shows a minute of GC activity when pooling is off, notice there are no promotions into Gen 2, and the size of the Gen 2 heap doesn't change. Which image do you want your game to produce? ;)
There is no way around the Garbage Collector's CPU "tax". Normally, you pay in lots of small increments spread over time, the same way real taxes are deducted from your paycheck. Following that analogy, using the ObjectPooler is like switching to the quarterly "lump sum" payments you make to the IRS if you are self-employed. You're still paying the same amount (mostly; the money analogy would need "transaction fees" and "interest rates"), it's just distributed differently.
In traditional applications, spending one or two full seconds in the GC in certain situations might be a better solution, but 99% of the time, games can't afford that kind of "lump sum" payment. Your game will run faster for a little while if you take a detour around the GC and use the ObjectPooler, but saying it's faster is like saying you don't have to pay taxes anymore because you're self-employed. Eventually, Uncle Sam will come calling, and he may have to garnish your wages.
03/10/2007 (9:04 am)
Thanks for the quick response, Stephen. What other problems did pooling solve? Are you sure pooling is really what fixed the hitching problem? To satisfy my own curiosity, I decided to bite the bullet and write a quick test app to profile the ObjectPooler. I created a class that implements ObjectPooler.IResetable. The class copies data from a static MemoryStream into its own collection to simulate actual work and use a little CPU and memory. In Game.Update I create some of these objects and have them process their data. In pooling mode, 80% of these object are randomly recycled into the pool.
The app toggles pooling once per minute and logs the average and max frame times for the last 60 seconds. Here's a piece of the log file. Avg and Max are frame timings in milliseconds, and the best and worst individual frame times are excluded from the average.
I also used perfmon (the GC counters are in the ".Net CLR Memory" section) and took some screen caps of its output. In both images, yellow is the number of bytes promoted from Gen 0 to Gen 1, blue is the number of bytes promoted from Gen 1 to Gen 2, skinny red is the Gen 1 heap in bytes, and thick red is the Gen 2 heap in bytes (note, the Gen 2 heap is drawn at a smaller scale, it's much larger than the Gen 1 heap).
The first image shows one minute of data with pooling turned on. Notice how much data is getting promoted to Gen 1 and Gen 2. At the end, promotions from Gen 1 increase the Gen 2 heap to hundreds of MB, which is then collected. That mammoth collection is the cause of the 700 ms Max frame time.
The second image shows a minute of GC activity when pooling is off, notice there are no promotions into Gen 2, and the size of the Gen 2 heap doesn't change. Which image do you want your game to produce? ;)
There is no way around the Garbage Collector's CPU "tax". Normally, you pay in lots of small increments spread over time, the same way real taxes are deducted from your paycheck. Following that analogy, using the ObjectPooler is like switching to the quarterly "lump sum" payments you make to the IRS if you are self-employed. You're still paying the same amount (mostly; the money analogy would need "transaction fees" and "interest rates"), it's just distributed differently.
In traditional applications, spending one or two full seconds in the GC in certain situations might be a better solution, but 99% of the time, games can't afford that kind of "lump sum" payment. Your game will run faster for a little while if you take a detour around the GC and use the ObjectPooler, but saying it's faster is like saying you don't have to pay taxes anymore because you're self-employed. Eventually, Uncle Sam will come calling, and he may have to garnish your wages.
#3
03/12/2007 (4:51 am)
Hmmm. Interesting, have you tried this test on 360?
#4
Since then, we haven't seen great needs for it and generally don't turn it on. When we do turn it on, though, we tend to see modest gains on the 360 (sorry no numbers, word of mouth from other members of the team). Remember that the 360 use the non-generational compact framework so a lot of the issues you point out above don't really apply (everything is gen2). Indeed, the number of garbage collections will be directly proportional to the amount of allocation you do, so pooling will reduce the number of garbage collections. I also think that you may not have picked the best test cases for pooling. I think that most objects that are pooled will already be gen2 when they are pooled, so there won't be any extra promotion (just some extra gen2 objects hanging around).
But with all that said, I'm not opposed to pulling the feature at some point. I just don't think that point is now. As stated in our docs, it is still unclear to us how valuable the feature is. We're aware of the issues you point out but aren't yet convinced it isn't important in some cases. In fact, I would still guess (you added ?? above to the word guess, but there is nothing wrong with an informed guess) that projectiles and explosions should use the object pooler. Although object pooling is certainly a bad memory management technique for all your objects in .NET, I haven't seen anyone claim it's not ever useful.
03/12/2007 (9:21 am)
@Paul, thanks for your feedback. What you say doesn't land on deaf ears. We put object pooling in fairly early on thinking it was going to be important. Usually we don't add a feature like that before seeing the need for it, but we have a lot of experience suggesting the need for object pooling so it was made part of the object model design. It was also thought that if we don't do it up front we'll never be able to add it later. Of course, the main reason for using object pooling in unmanaged enviroments is to avoid memory fragmentation, which isn't an issue here, so our previous experience is perhaps not relevant.Since then, we haven't seen great needs for it and generally don't turn it on. When we do turn it on, though, we tend to see modest gains on the 360 (sorry no numbers, word of mouth from other members of the team). Remember that the 360 use the non-generational compact framework so a lot of the issues you point out above don't really apply (everything is gen2). Indeed, the number of garbage collections will be directly proportional to the amount of allocation you do, so pooling will reduce the number of garbage collections. I also think that you may not have picked the best test cases for pooling. I think that most objects that are pooled will already be gen2 when they are pooled, so there won't be any extra promotion (just some extra gen2 objects hanging around).
But with all that said, I'm not opposed to pulling the feature at some point. I just don't think that point is now. As stated in our docs, it is still unclear to us how valuable the feature is. We're aware of the issues you point out but aren't yet convinced it isn't important in some cases. In fact, I would still guess (you added ?? above to the word guess, but there is nothing wrong with an informed guess) that projectiles and explosions should use the object pooler. Although object pooling is certainly a bad memory management technique for all your objects in .NET, I haven't seen anyone claim it's not ever useful.
#5
Compact framework team blog
03/13/2007 (10:26 am)
Someone just forwarded me this link, which I thought was relevant to this discussion:Compact framework team blog
#6
You're right about my test case, too. I've been tinkering with the test app off and on all week. The latest version gives each object a random lifespan within a range, and objects are processed every frame while alive. I also create two different kinds of objects, long-term and short-term. There are fewer long-term objects, but they are larger and have longer lifespans, while only short-term objects are pooled. I use a seeded instance of the Random class to generate all the variations, and it gets re-seeded between runs so both pooling and non-pooling mode are using the exact same inputs.
I have about 15 configurable settings in my app.config file, and I must have tried dozens of different configurations on three different PCs that range from low-end to high-end, and I still haven't been able to find a single case where pooling gives better performance by any measure. I also tried a version that used T2DSprites instead of my test object, but it didn't have any effect on the relative performance of pooling.
I hope I don't sound like a grumpy contrarian trying to poke holes in your shiny new engine. TorqueX is an awesome library, even at this early beta stage. Pooling struck me as C++ throwback, but I'm a pragmatist, and will use what works. If there is a way to improve performance with the ObjectPooler, I couldn't find it.
I've found many ways to bring the GC to its knees with pooling, but the best-case scenario seems to be identical performance. In the end, I think object pooling in .Net is essentially "anti-Chicken Soup"; It might not hurt, but it sure can't help ;)
03/18/2007 (8:18 am)
Thanks for the detailed response, Clark. My game doesn't really translate to the 360, so I haven't been considering its implications. I didn't even know it had a non-generational GC until I read your post.You're right about my test case, too. I've been tinkering with the test app off and on all week. The latest version gives each object a random lifespan within a range, and objects are processed every frame while alive. I also create two different kinds of objects, long-term and short-term. There are fewer long-term objects, but they are larger and have longer lifespans, while only short-term objects are pooled. I use a seeded instance of the Random class to generate all the variations, and it gets re-seeded between runs so both pooling and non-pooling mode are using the exact same inputs.
I have about 15 configurable settings in my app.config file, and I must have tried dozens of different configurations on three different PCs that range from low-end to high-end, and I still haven't been able to find a single case where pooling gives better performance by any measure. I also tried a version that used T2DSprites instead of my test object, but it didn't have any effect on the relative performance of pooling.
I hope I don't sound like a grumpy contrarian trying to poke holes in your shiny new engine. TorqueX is an awesome library, even at this early beta stage. Pooling struck me as C++ throwback, but I'm a pragmatist, and will use what works. If there is a way to improve performance with the ObjectPooler, I couldn't find it.
I've found many ways to bring the GC to its knees with pooling, but the best-case scenario seems to be identical performance. In the end, I think object pooling in .Net is essentially "anti-Chicken Soup"; It might not hurt, but it sure can't help ;)
#7
On the contrary, it's absolutely appreciated. If we could verify that pooling will never help I'd love to removed the concept from the engine. It adds some complexity in a couple places and can lead to some errors (e.g., not properly reseting fields can now result in subtle bugs). But the idea that if might be crucial in some cases (in particular on the 360) has lead us to keep it around. Still, I think you make a good case to not use it except in very specific cases (and you haven't found any of those).
One thing about the tests you said you did: You mentioned that only short term objects are pooled. I think you might want to flip that because short term objects (in standard framework) will benefit the least from being pooled. In particular, the case I see as being potentially beneficial is the projectile case, where you have maybe 20 instances alive at any one time, but generating, say 20 every second (and therefore killing 20 every second), and when the projectile goes away, it will arleady be gen2. In that case, just having a pool of 20 projectiles seems like an obvious win. But, of course, it bears testing.
03/18/2007 (10:01 am)
@PaulQuote:I hope I don't sound like a grumpy contrarian trying to poke holes in your shiny new engine.
On the contrary, it's absolutely appreciated. If we could verify that pooling will never help I'd love to removed the concept from the engine. It adds some complexity in a couple places and can lead to some errors (e.g., not properly reseting fields can now result in subtle bugs). But the idea that if might be crucial in some cases (in particular on the 360) has lead us to keep it around. Still, I think you make a good case to not use it except in very specific cases (and you haven't found any of those).
One thing about the tests you said you did: You mentioned that only short term objects are pooled. I think you might want to flip that because short term objects (in standard framework) will benefit the least from being pooled. In particular, the case I see as being potentially beneficial is the projectile case, where you have maybe 20 instances alive at any one time, but generating, say 20 every second (and therefore killing 20 every second), and when the projectile goes away, it will arleady be gen2. In that case, just having a pool of 20 projectiles seems like an obvious win. But, of course, it bears testing.
#8
I think we may using different definitions for short-term and long-term. I think when you say short-term objects, you mean it in the .Net sense, i.e. objects that are used in a single line to code to hold return values and what not. If so, yes, pooling those kind of objects is pointless.
When I say short-term, I'm using it in a game sense, which would include projectiles and explosions.
As a point of reference, the current config settings for my test app give short-term objects a lifetime of between 1 and 5 seconds, and long-term objects live from 15 to 45 seconds. Even that isn't very long. Earlier today, I played a game of Supreme Commander, and most of my buildings and construction units had a lifespan of at least 45 minutes.
I think the last few series of tests I ran are EXACTLY the kind of tests you asked for in the last post.
I'm looking over the log files now, and I can see that between 15 and 25 short-term "projectile-like" objects are both created and destroyed in each frame. The following log entries are an excellent summation of the results I got. You can see that pooling was only 1/10 of a millisecond slower on average, but it's max value was almost 20ms higher, which is enough to cause a noticeable hitch. Pooling is usually very close in average frame time, but it's max time is often noticeably higher.
The first line of each entry is a cumulative look at two minutes of profiling, but the second line (with object counts) represents a snapshot of a single frame.
FYI, I made some manual edits to the log data after posting because I accidentally copied the wrong entry from my log file.
03/18/2007 (12:08 pm)
I understand. This is an amorhphous subject that is hard to quantify without serious inspection. I wouldn't take it out until you get some more solid data on the 360. I'm only worried about my game, but you have to worry about every game that uses TorqueX.I think we may using different definitions for short-term and long-term. I think when you say short-term objects, you mean it in the .Net sense, i.e. objects that are used in a single line to code to hold return values and what not. If so, yes, pooling those kind of objects is pointless.
When I say short-term, I'm using it in a game sense, which would include projectiles and explosions.
As a point of reference, the current config settings for my test app give short-term objects a lifetime of between 1 and 5 seconds, and long-term objects live from 15 to 45 seconds. Even that isn't very long. Earlier today, I played a game of Supreme Commander, and most of my buildings and construction units had a lifespan of at least 45 minutes.
I think the last few series of tests I ran are EXACTLY the kind of tests you asked for in the last post.
I'm looking over the log files now, and I can see that between 15 and 25 short-term "projectile-like" objects are both created and destroyed in each frame. The following log entries are an excellent summation of the results I got. You can see that pooling was only 1/10 of a millisecond slower on average, but it's max value was almost 20ms higher, which is enough to cause a noticeable hitch. Pooling is usually very close in average frame time, but it's max time is often noticeably higher.
The first line of each entry is a cumulative look at two minutes of profiling, but the second line (with object counts) represents a snapshot of a single frame.
Pooling:False Avg:22.61 ms Max:29.67 ms Min:14.67 ms Last:20.54 ms Objects:643 Long:150/150 Short:493/500 Pooled:0/0 Created:19 Removed:23 Pooling:True Avg:22.72 ms Max:56.07 ms Min:13.71 ms Last:22.00 ms Objects:635 Long:150/150 Short:485/500 Pooled:3/10 Created:16 Removed:15
FYI, I made some manual edits to the log data after posting because I accidentally copied the wrong entry from my log file.
#9
This is probably a good opportunity to point out another problem with object pooling. This is that as objects get pooled, their components and other contained objects start to become detached from them in memory, so you lose locality of reference. To improve this situation we include a PoolWithComponents method, but that only handles the components. This is yet another reason I'd be happy to pull the whole feature.
03/18/2007 (12:32 pm)
@Paul, that's great info. I agree that your short-term case is much like a projectile, so it's a good test. I wonder if you wouldn't mind sharing your test app with us. I'd like to run it on the 360 to see what kind of results we get there. Heck, maybe it will provide justification for pulling object pooling altogether. :)This is probably a good opportunity to point out another problem with object pooling. This is that as objects get pooled, their components and other contained objects start to become detached from them in memory, so you lose locality of reference. To improve this situation we include a PoolWithComponents method, but that only handles the components. This is yet another reason I'd be happy to pull the whole feature.
#10
I tried to compile it in a 360 project after reading your post from earlier today, and it won't be a trivial port. The library isn't targeted at gaming. I've used it in Windows Forms, ASP.Net, and Windows Service applications in the past, and that means it references tons of stuff that isn't part of the .NET Compact Framework.
What do you use for configuration and logging on .Net CF? Even though I've added a lot of custom stuff, it all sits on top of the standard .Net classes. For example, my logging code won't work without the Trace support in System.Dianostics, and my configuration classes won't work without the Configuration Manager in the System.Configuration namespace, both of which seem to be absent on the 360.
I could try and pull my library out of the test app, but the way I nest, accumulate and process the profiling data is 50% of the app, and all the heavy lifting in that department is done by the logging classes in my library. Let me take a closer look and get back to you. I may be over-estimating the difficulty since I've never really thought about this before.
03/18/2007 (1:32 pm)
Here's the thing. My test app uses my own .Net library as well as TorqueX. I tried to compile it in a 360 project after reading your post from earlier today, and it won't be a trivial port. The library isn't targeted at gaming. I've used it in Windows Forms, ASP.Net, and Windows Service applications in the past, and that means it references tons of stuff that isn't part of the .NET Compact Framework.
What do you use for configuration and logging on .Net CF? Even though I've added a lot of custom stuff, it all sits on top of the standard .Net classes. For example, my logging code won't work without the Trace support in System.Dianostics, and my configuration classes won't work without the Configuration Manager in the System.Configuration namespace, both of which seem to be absent on the 360.
I could try and pull my library out of the test app, but the way I nest, accumulate and process the profiling data is 50% of the app, and all the heavy lifting in that department is done by the logging classes in my library. Let me take a closer look and get back to you. I may be over-estimating the difficulty since I've never really thought about this before.
#11
03/19/2007 (9:13 am)
Roger that. 
Torque 3D Owner Stephen Zepp
Choosing to use it is an option you have to select, it's not on by default.