Memory usage tools for Torque
by Peter Simard · in Torque Game Engine · 04/03/2008 (12:35 pm) · 14 replies
Our game is currently using over 1Gig of RAM on most systems. I have been searching around the site and on Google for tools to analyze the memory usage, but haven't come up with anything.
I have read the documentation on the DEBUG_GUARD option, but it only seems useful for detecting leaks, not reporting where the memory is being allocated. Is there a way to use it that will report where all the memory is currently allocated to?
Perhapse the oddest thing, is that my development system only uses around 400MB of RAM, while most other systems use well over 1Gig. What could cause such a divergence? Prefs files are the same.
Any help on this would be very much appreciated.
I have read the documentation on the DEBUG_GUARD option, but it only seems useful for detecting leaks, not reporting where the memory is being allocated. Is there a way to use it that will report where all the memory is currently allocated to?
Perhapse the oddest thing, is that my development system only uses around 400MB of RAM, while most other systems use well over 1Gig. What could cause such a divergence? Prefs files are the same.
Any help on this would be very much appreciated.
#2
04/03/2008 (2:47 pm)
My system and some of the other machines have 512MB graphics cards, so that doesn't seem to be the factor. 
#3
04/03/2008 (3:24 pm)
I posted this in the TGEA forums. I wonder if it fits your situation.Quote:We load lotsa of big things into RAM in our projects, so it was never a surprise to see the thing head upwards to a GB. That is until we tried turning off the Torque Memory Manager and found our actual usage was more like 400MB. So given that less is more, we went with the easy solution and turned it off.
That is until I had to debug something recently and in debug mode it ballooned to more than 2GB for our latest TGEA product, making it difficult to debug. In fact it usually crashes with some assert. Long story short: Without the mem manager it uses only 520MB. With it allocates over 130 pages of memory.
So then I tried the next obvious step and increased the MinPageSize from 8MB up to 32MB. Now it only allocates a new page once and it runs at a svelt 540MB. (I tried 8, 16, 32, 48 & 64 where 32 gave me te best results.) Oh, and there's no more asserting going on.
Now I know doesn't release what it allocates, but does it ever attempt to reuse what it's got? It seems to be just wasting it.
#4
04/03/2008 (3:37 pm)
@James - Thanks for your response. To be clear, did you end up disabling TMM or just increasing the MinPageSize to 32?
#5
Look in platformMemory.cpp (or .cc for TGE) for the console function "dumpMemSnapshot"
That dumps all allocated memory to a file. I've used Excel to then organize the data for different analysis.
This dump file put my memory at around 80 Meg. Looking on the Windows Task Manager, the figure is roughly four to five times that. The 80 Meg number is what I would expect for the assets I have loaded. I'm not sure why the task manager number is so much larger. (This was for both debug and release, with debug about 40 Megs higher).
Good luck and I'm interested in what you find and use.
04/03/2008 (5:25 pm)
This might also be useful.Look in platformMemory.cpp (or .cc for TGE) for the console function "dumpMemSnapshot"
That dumps all allocated memory to a file. I've used Excel to then organize the data for different analysis.
This dump file put my memory at around 80 Meg. Looking on the Windows Task Manager, the figure is roughly four to five times that. The 80 Meg number is what I would expect for the assets I have loaded. I'm not sure why the task manager number is so much larger. (This was for both debug and release, with debug about 40 Megs higher).
Good luck and I'm interested in what you find and use.
#6
I believe that if you can find a way to change the page size to something that works for you, then that would be a better solution than disabling the TMM.
We've done so many changes to our system, I'd be very interested to hear if this helped anyone else.
04/03/2008 (11:21 pm)
We first disabled the TMM to get an "accurate" read on how much RAM we were using according to Windows. Then when I realized that changing the page size helped bring the TMM-Enabled size back to a reasonable level, I tried different sizes until I found the one that worked best for that particular project, which was 32.I believe that if you can find a way to change the page size to something that works for you, then that would be a better solution than disabling the TMM.
We've done so many changes to our system, I'd be very interested to hear if this helped anyone else.
#7
Each person loaded the same mission, perfermed a "dumpMemSnapshot" and reported the memory usage from the task manager. Here are the results:
There are several things that stand out:
1) Torque is allocating the same amount of RAM for each person (around 159MB)
2) Memory usage as reported by task manager seems inconsistent
3) Some people reported less memory used than Torque allocated
What would cause some people to use over 1GB of RAM, while others use much less?
Are there tools useful for obtaining a more accurate snapshot of RAM usage?
04/04/2008 (12:01 pm)
I have been having people run tests on their PCs to get some data on memory usuage. Each person has an identical copy of the game, and deleted their prefs files. Each person loaded the same mission, perfermed a "dumpMemSnapshot" and reported the memory usage from the task manager. Here are the results:
console: total memory used: 158970764 task manager: 1,054,769 K OS: Windows XP Professional x64 SP2 CPU: Intel Core2 6600 @ 2.4GHz RAM: 2 GB Video: ATI FireGl V5100 128MB (driver 8.323.0.0 12/6/2006)
Console: 159100604 Task manager: 120540k OS: Windows XP Professional SP2 CPU: PentiumD-3Ghz RAM: 1 GB Video: GeForce 7300GT 256MB
console: total memory used 159106480 task manager: 1,040,660 OS: XP home edition version 2002 SP2 cpu: core 2 2.13 ghz ram: 3 gigs ddr2 800 video card: nvidia 512 megs video memory
Console: 158,662,852 Task manager: 88,340k OS: Windows XP Professional SP2 CPU: INtel Core2 CPU @ 2.13GHz RAM: 1 GB Video: Radeon X1900 GT
console: total memory used: 159414392 task manager: 1,039,036 K OS: Windows XP Home Edition CPU: Intel Core 2 6700 2.6GHz RAM: 2 GB Video: GeForce 8800 GTX
console: total memory used: 158624392 task mananger: 433,232K OS: Windows XP Home CPU: Intel Core2 2.6 RAM: 2GB Video: GeForce 8800GTX
There are several things that stand out:
1) Torque is allocating the same amount of RAM for each person (around 159MB)
2) Memory usage as reported by task manager seems inconsistent
3) Some people reported less memory used than Torque allocated
What would cause some people to use over 1GB of RAM, while others use much less?
Are there tools useful for obtaining a more accurate snapshot of RAM usage?
#8
04/04/2008 (12:15 pm)
As far as I am aware, Task Manager treats memory that's "active" differently from memory that is currently off-paged to disk (I could be wrong here). To Torque however, it's just memory--we don't care where the OS has it currently stored.
#9
There seems to be a huge discrepancy between what Torque is allocating, and what Windows is.
04/04/2008 (1:20 pm)
@Stephen - Nonetheless, Windows seems to be allocating over a gig of RAM on most systems. We are getting frequent reports of timeouts during loading, and unplayable framerates, on machines that should be able to handle it.There seems to be a huge discrepancy between what Torque is allocating, and what Windows is.
#10
I see Torque reporting 158-159M for most of your clients (give or take), and all of the task manager reports fluctuating wildly--anywhere from 88M to 1G.
04/04/2008 (1:47 pm)
Either you or I are reading those numbers incorrectly, hehe.I see Torque reporting 158-159M for most of your clients (give or take), and all of the task manager reports fluctuating wildly--anywhere from 88M to 1G.
#11
I realize you aren't asking for specifics, but tools to help you figure out what's going on, and I respect that. Unfortunately, it's very difficult to help you analyze what the issue is without a really deep understanding of your project.
I can comment that usually, high memory footprints come from unoptimized assets (large textures when they aren't needed, over-developed models (I mean in poly count!), and in general just having too much "in game" at once--too big/full of a mission effectively, that might be broken out into separate missions, or dynamically paged/unpaged from the running mission?
Keep in mind that during startup, it is (default anyway) pre-loading every single asset from every datablock.
04/04/2008 (2:00 pm)
Follow up:I realize you aren't asking for specifics, but tools to help you figure out what's going on, and I respect that. Unfortunately, it's very difficult to help you analyze what the issue is without a really deep understanding of your project.
I can comment that usually, high memory footprints come from unoptimized assets (large textures when they aren't needed, over-developed models (I mean in poly count!), and in general just having too much "in game" at once--too big/full of a mission effectively, that might be broken out into separate missions, or dynamically paged/unpaged from the running mission?
Keep in mind that during startup, it is (default anyway) pre-loading every single asset from every datablock.
#12
Run -> perfmon -> Enter
Please see my blog post about this:
www.garagegames.com/blogs/4517/11311
04/05/2008 (1:11 am)
The task manager is a wholly inaccurate and bad estimate of memory usage. If you want to get a better picture of memory usage your first step should be to stop looking at the task manager and start using the performance monitor. It is built into XP and Vista.Run -> perfmon -> Enter
Please see my blog post about this:
www.garagegames.com/blogs/4517/11311
#13
04/06/2008 (11:58 am)
It has quite something to do with the graphics driver version. Different driver versions handle the hyper memory / turbo cache differently. Especially on ATI this can "fixed allocate" the little nothing of 1536MB - real VRAM on 2GB Systems 
#14
I've replaced Taskman.exe with it, in fact. Good luck convincing Windows to take it, though ;)
On Linux, Valgrind is sometimes useful to look for memory leaks etc., but Torque is slowed down
immensely while debugging this way, so it's best to run on a really fast computer and use some
automated testing scripts, rather than an input-reliant game.
Most operating systems aren't showing more than a user-friendly total of memory usage, so if you
happen to have 512MB of graphics memory and the desktop runs a 3D desktop, you could in some
cases see that the system process looks like it gobbles memory at an alarming rate.
Look for a "leak checker" or some other sort of memory debugger. If you do find a helpful one,
please post back here, as I'm currently covering four platforms, and on two I need more tools!
04/06/2008 (3:40 pm)
Process Explorer also gives you a very detailed overview, compared to the regular Taskmanager.I've replaced Taskman.exe with it, in fact. Good luck convincing Windows to take it, though ;)
On Linux, Valgrind is sometimes useful to look for memory leaks etc., but Torque is slowed down
immensely while debugging this way, so it's best to run on a really fast computer and use some
automated testing scripts, rather than an input-reliant game.
Most operating systems aren't showing more than a user-friendly total of memory usage, so if you
happen to have 512MB of graphics memory and the desktop runs a 3D desktop, you could in some
cases see that the system process looks like it gobbles memory at an alarming rate.
Look for a "leak checker" or some other sort of memory debugger. If you do find a helpful one,
please post back here, as I'm currently covering four platforms, and on two I need more tools!
Torque Owner Thomas Oliver