TGB ImageMap Datablock manager (pure script or engine enhancement)
by William Hilke · 07/03/2006 (11:19 am) · 10 comments
Download Code File
For a project I am working on I needed a method to control when ImageMaps got loaded into memory, and when they got unloaded. I also wanted to display loading progress to my users so that they didn't think the game had froze up on them. For my class to work you need to make sure any imagemaps that you want this class to control are not set to preload. You can also set them to allowUnload, if you want them to get removed from memory; like say if your switching to a new level and don't need the previous levels image data.
Instructions to use the class are provided in the example script. Please let me know if you see any problems, or know a better way to do this.
Update:
The pure script solution creates a hidden static sprite for each imagemap that it loads. If you want to remove this overhead, I've included a module to compile into the engine. Just add DynamicImageLoader.cc to your project and recompile TGB. The TorqueScript class will detect if you have compiled in the engine support and use that instead of the static sprites.
Update (Oct 7th):
Thanks to DooGoG's keen eye I fixed DynamicImagLloader.cs (line #152). This should fix the callback.
Update (6/3/07):
I just tested this resource against TGB v1.5 Beta 3 and it works as packaged.
For a project I am working on I needed a method to control when ImageMaps got loaded into memory, and when they got unloaded. I also wanted to display loading progress to my users so that they didn't think the game had froze up on them. For my class to work you need to make sure any imagemaps that you want this class to control are not set to preload. You can also set them to allowUnload, if you want them to get removed from memory; like say if your switching to a new level and don't need the previous levels image data.
Instructions to use the class are provided in the example script. Please let me know if you see any problems, or know a better way to do this.
Update:
The pure script solution creates a hidden static sprite for each imagemap that it loads. If you want to remove this overhead, I've included a module to compile into the engine. Just add DynamicImageLoader.cc to your project and recompile TGB. The TorqueScript class will detect if you have compiled in the engine support and use that instead of the static sprites.
Update (Oct 7th):
Thanks to DooGoG's keen eye I fixed DynamicImagLloader.cs (line #152). This should fix the callback.
Update (6/3/07):
I just tested this resource against TGB v1.5 Beta 3 and it works as packaged.
#2
Correction: I think in line 152, "mCallbackFunc" should be "%this.mCallbackFunc"
10/03/2006 (2:01 am)
Thanks much for this great resource!Correction: I think in line 152, "mCallbackFunc" should be "%this.mCallbackFunc"
#3
01/14/2007 (2:49 am)
Great resource, will definitely help to get rid of some "interesting" problems on my old testing system :) 
#4
I have couple of questions:
1. Why do we allow more than one instance of each t2dImageMapDatablock to be loaded? Aren't these definitions, and the sprites are the instanced objects that we need to worry about?
If the Add() method checked the mDatablocks SimSet for the datablock, then Remove() wouldn't have to loop so much.
2. Remove() looks buggy in the ScriptOnly version.
The For Loop uses %this.mImageCount for the upper index bound...but you change the SimSet in the loop as imageMaps are found.
These lines seem problematic to me:
%spr = %this.mImages.getObject(%i);
%this.mImages.remove(%spr);
As you loop through and remove items from the SimSet, the objects after that index inside all shift down by 1. So if you have 5 elements, say:
%this.mImages.getObject(0) = "A"
%this.mImages.getObject(1) = "A"
%this.mImages.getObject(2) = "B"
%this.mImages.getObject(3) = "A"
%this.mImages.getObject(4) = "A"
as you loop through and delete "A" the first time (%i is 0), the SimSet then looks like:
%this.mImages.getObject(0) = "A"
%this.mImages.getObject(1) = "B"
%this.mImages.getObject(2) = "A"
%this.mImages.getObject(3) = "A"
the second iteration SimSet looks like this (%i is 1) - it skips the first "A" and leaves it in the set.
%this.mImages.getObject(0) = "A"
%this.mImages.getObject(1) = "B"
%this.mImages.getObject(2) = "A"
%this.mImages.getObject(3) = "A"
the third iteration SimSet looks like this (%i is 2) - it deletes the second "A"
%this.mImages.getObject(0) = "A"
%this.mImages.getObject(1) = "B"
%this.mImages.getObject(2) = "A"
For the next iteration, %i is 3, which is now greater than the number of elements, so it doesn't execute the loop...leaving 2 "A" elements in there.
I think the reason this script works for you is because you have the c++ methods in your build, so it bypasses this way of deletion and simply calls:
%imgMap.UnloadImageMap();
which should work fine.
My point is that the ScriptOnly version may not be functioning and you've envisioned. I am not at my workstation right now or I would test out my theories here. I could be totally off.
All of this goes back to #1 - do we need to store multiple references to the same Datablock? or is 1 enough? If 1 is enough, then the delete can be much simpler.
I love these kinds of articles - great job. These are the types of discussions that are so helpful to this community.
Thanks,
Aaron
01/15/2008 (12:37 pm)
William - excellent work!I have couple of questions:
1. Why do we allow more than one instance of each t2dImageMapDatablock to be loaded? Aren't these definitions, and the sprites are the instanced objects that we need to worry about?
If the Add() method checked the mDatablocks SimSet for the datablock, then Remove() wouldn't have to loop so much.
2. Remove() looks buggy in the ScriptOnly version.
The For Loop uses %this.mImageCount for the upper index bound...but you change the SimSet in the loop as imageMaps are found.
These lines seem problematic to me:
%spr = %this.mImages.getObject(%i);
%this.mImages.remove(%spr);
As you loop through and remove items from the SimSet, the objects after that index inside all shift down by 1. So if you have 5 elements, say:
%this.mImages.getObject(0) = "A"
%this.mImages.getObject(1) = "A"
%this.mImages.getObject(2) = "B"
%this.mImages.getObject(3) = "A"
%this.mImages.getObject(4) = "A"
as you loop through and delete "A" the first time (%i is 0), the SimSet then looks like:
%this.mImages.getObject(0) = "A"
%this.mImages.getObject(1) = "B"
%this.mImages.getObject(2) = "A"
%this.mImages.getObject(3) = "A"
the second iteration SimSet looks like this (%i is 1) - it skips the first "A" and leaves it in the set.
%this.mImages.getObject(0) = "A"
%this.mImages.getObject(1) = "B"
%this.mImages.getObject(2) = "A"
%this.mImages.getObject(3) = "A"
the third iteration SimSet looks like this (%i is 2) - it deletes the second "A"
%this.mImages.getObject(0) = "A"
%this.mImages.getObject(1) = "B"
%this.mImages.getObject(2) = "A"
For the next iteration, %i is 3, which is now greater than the number of elements, so it doesn't execute the loop...leaving 2 "A" elements in there.
I think the reason this script works for you is because you have the c++ methods in your build, so it bypasses this way of deletion and simply calls:
%imgMap.UnloadImageMap();
which should work fine.
My point is that the ScriptOnly version may not be functioning and you've envisioned. I am not at my workstation right now or I would test out my theories here. I could be totally off.
All of this goes back to #1 - do we need to store multiple references to the same Datablock? or is 1 enough? If 1 is enough, then the delete can be much simpler.
I love these kinds of articles - great job. These are the types of discussions that are so helpful to this community.
Thanks,
Aaron
#5
Thanks for the comments. As you can see I posted this over a year and a half ago, and to be honest I haven't been playing around with TGB lately :)
However, I will try and answer your questions. Note: This is based solely on my observations of the engine code, and hopefully the engine hasn't changed in these areas since I wrote this, otherwise all bets are off.
I'll first describe how the TGB resource loader works in relation to imagemap datablocks and sprites (in case you are not aware), and then I'll talk about the "Preload" and "AllowUnload" properties of the imagemap datablock.
First, by default (Preload set to true), the image associated with an imagemap gets loaded into memory when that imagemap gets created. This creation happens when the script processes that block of script code. This means that the image is not tied to any sprite object directly; which also means that you can have multiple sprite objects that use the same image and end up with only one image copy in memory. Also by default (AllowUnload set to false), the imagemap datablock never gets unloaded (even if you delete all sprites that use that imagemap).
Second, "Preload". If this property is set to false then the engine does not load the image into memory when you create an instance of the imagemap. It'll delay the loading until you actually need it. One way the engine knows you need it is when you create a staticsprite object that uses an imagemap block with preload=false and it hasn't loaded the image data yet. "AllowUnload" works in reverse. As soon as the last reference to a imagemap object (all sprites that use that imagemap) is deleted and that property is set, it'll unload the data from memory.
So, with all that in mind I'll answer your questions now :)
1.) t2dImageMapDatablock are the objects that describe a specific image. So yes, they are definitions in a sense but they are still just script objects under the hood. This is why I only care about datablocks, and not the sprites in your code that use them. My system is intended for the loader class to get deleted only after you have deleted all your sprite objects first.
1.a) The pure-script version of my system forces a t2dImageMapDatablock (with its Preload property set to false) to load the image into memory by creating a hidden sprite for each imagemap block added to the system. When this hidden sprite gets created the image gets loaded into memory. This is also why the hidden sprites don't get created until the Load() method is called.
1.b) You are correct that the system allows the same t2dImageMapDatablock to be added more then once, and this probably isn't something you want to do. I should update the code to not allow this, but this isn't why Remove does what it does.
2.) I think you are missing one crucial statement in the Remove() method. Line #80 : return true;. The Remove() method is designed to only remove one item from the SimSet. The reason why it loops through the mDatablocks SimSet and finds the imagemap that matches the correct index is because in the script only version it needs to first remove the static hidden sprite before removing the datablock. The two SimSets are matched by the same index. So finding the index in the mDatablocks simset, I also know the index into the mImages SimSet that contains the hidden sprite that links to that imagemap.
2.a) I know there are better ways to tie the hidden sprite with the imagedata block, but understand that I wrote this resource when I was still pretty new to TorqueScript :) Maybe I'll rewrite this resource to be a bit more efficient.
2.b) The use of the Remove() method is so that you can have granular control of when you unload images from memory. You first need to make sure to delete all instances of any sprites that use that imagemap, then you can call Remove and pass in that imagemap. if you call Delete() on the class it'll do the same as Remove(), but for all imagemaps it knows about.
I hope this gives you some more insight into the system, and maybe answer your questions. Let me know if it doesn't :)
William
01/15/2008 (11:51 pm)
Aaron,Thanks for the comments. As you can see I posted this over a year and a half ago, and to be honest I haven't been playing around with TGB lately :)
However, I will try and answer your questions. Note: This is based solely on my observations of the engine code, and hopefully the engine hasn't changed in these areas since I wrote this, otherwise all bets are off.
I'll first describe how the TGB resource loader works in relation to imagemap datablocks and sprites (in case you are not aware), and then I'll talk about the "Preload" and "AllowUnload" properties of the imagemap datablock.
First, by default (Preload set to true), the image associated with an imagemap gets loaded into memory when that imagemap gets created. This creation happens when the script processes that block of script code. This means that the image is not tied to any sprite object directly; which also means that you can have multiple sprite objects that use the same image and end up with only one image copy in memory. Also by default (AllowUnload set to false), the imagemap datablock never gets unloaded (even if you delete all sprites that use that imagemap).
Second, "Preload". If this property is set to false then the engine does not load the image into memory when you create an instance of the imagemap. It'll delay the loading until you actually need it. One way the engine knows you need it is when you create a staticsprite object that uses an imagemap block with preload=false and it hasn't loaded the image data yet. "AllowUnload" works in reverse. As soon as the last reference to a imagemap object (all sprites that use that imagemap) is deleted and that property is set, it'll unload the data from memory.
So, with all that in mind I'll answer your questions now :)
1.) t2dImageMapDatablock are the objects that describe a specific image. So yes, they are definitions in a sense but they are still just script objects under the hood. This is why I only care about datablocks, and not the sprites in your code that use them. My system is intended for the loader class to get deleted only after you have deleted all your sprite objects first.
1.a) The pure-script version of my system forces a t2dImageMapDatablock (with its Preload property set to false) to load the image into memory by creating a hidden sprite for each imagemap block added to the system. When this hidden sprite gets created the image gets loaded into memory. This is also why the hidden sprites don't get created until the Load() method is called.
1.b) You are correct that the system allows the same t2dImageMapDatablock to be added more then once, and this probably isn't something you want to do. I should update the code to not allow this, but this isn't why Remove does what it does.
2.) I think you are missing one crucial statement in the Remove() method. Line #80 : return true;. The Remove() method is designed to only remove one item from the SimSet. The reason why it loops through the mDatablocks SimSet and finds the imagemap that matches the correct index is because in the script only version it needs to first remove the static hidden sprite before removing the datablock. The two SimSets are matched by the same index. So finding the index in the mDatablocks simset, I also know the index into the mImages SimSet that contains the hidden sprite that links to that imagemap.
2.a) I know there are better ways to tie the hidden sprite with the imagedata block, but understand that I wrote this resource when I was still pretty new to TorqueScript :) Maybe I'll rewrite this resource to be a bit more efficient.
2.b) The use of the Remove() method is so that you can have granular control of when you unload images from memory. You first need to make sure to delete all instances of any sprites that use that imagemap, then you can call Remove and pass in that imagemap. if you call Delete() on the class it'll do the same as Remove(), but for all imagemaps it knows about.
I hope this gives you some more insight into the system, and maybe answer your questions. Let me know if it doesn't :)
William
#6
This whole datablock manager concept is interesting to me. I'm glad you worked on this, and I would love to see a new version. Hopefully I'll have time to go back and work on some more framework stuff once this game ships in a couple of weeks...
Since I don't have the engine source (yet), would you be able to tell me a little about object texture memory allocation?
Does the datablock allocate memory when it gets loaded, or does the allocation happen when the sprite is added to the scene?
If I add 10 of the same sprites, does that add 10x the tex mem usage, or do the sprite instances get references to the source datablock texture?
Thanks,
Aaron
01/23/2008 (2:23 pm)
Thanks for the response William. I am sorry I took so long to respond. Working 18 hour days for 3 straight weeks is catching up to me ;)This whole datablock manager concept is interesting to me. I'm glad you worked on this, and I would love to see a new version. Hopefully I'll have time to go back and work on some more framework stuff once this game ships in a couple of weeks...
Since I don't have the engine source (yet), would you be able to tell me a little about object texture memory allocation?
Does the datablock allocate memory when it gets loaded, or does the allocation happen when the sprite is added to the scene?
If I add 10 of the same sprites, does that add 10x the tex mem usage, or do the sprite instances get references to the source datablock texture?
Thanks,
Aaron
#7
It depends on if you use the Preload property or not.
If Preload is set to true, the texture will get allocated as soon as the script engine creates the datablock object. If it is set to false, then the texture will get allocated as soon as a display object (sprite/tile/animation/etc) is created that has a valid datablock assigned. If you create a display object and don't assign a datablock, nothing will get allocated until you do the assignment. The display object doesn't have to be part of a scene for the allocation to happen.
ImageDatablocks only allocate their image data to memory once, no matter how many display objects reference it. However, If you have two ImageDatablock objects that use the same source image, I *think* it will allocate that image in memory twice (I'm not sure if the underlying resource manager lets that happen or not).
01/26/2008 (6:58 pm)
Aaron,It depends on if you use the Preload property or not.
If Preload is set to true, the texture will get allocated as soon as the script engine creates the datablock object. If it is set to false, then the texture will get allocated as soon as a display object (sprite/tile/animation/etc) is created that has a valid datablock assigned. If you create a display object and don't assign a datablock, nothing will get allocated until you do the assignment. The display object doesn't have to be part of a scene for the allocation to happen.
ImageDatablocks only allocate their image data to memory once, no matter how many display objects reference it. However, If you have two ImageDatablock objects that use the same source image, I *think* it will allocate that image in memory twice (I'm not sure if the underlying resource manager lets that happen or not).
#8
I've been battling the load time vs run time loading and it seems like the game "hitches" pretty bad when I load datablocks just before using them. It wouldn't be so bad if I could use preload = "0", but it appears that linked imagemaps and their referenced imagemaps all need to be preloaded or the game will crash. I've foolder around a bit with this and have fallen back on preloading datablocks for all linked imagemaps...which happens to be a lot in my case.
The game uses a ton of memory in this case, but dynamically adding/removing animations from the game is pretty seamless. The exec("foo.cs") call is the hitch.
01/27/2008 (8:54 am)
Thanks again William.I've been battling the load time vs run time loading and it seems like the game "hitches" pretty bad when I load datablocks just before using them. It wouldn't be so bad if I could use preload = "0", but it appears that linked imagemaps and their referenced imagemaps all need to be preloaded or the game will crash. I've foolder around a bit with this and have fallen back on preloading datablocks for all linked imagemaps...which happens to be a lot in my case.
The game uses a ton of memory in this case, but dynamically adding/removing animations from the game is pretty seamless. The exec("foo.cs") call is the hitch.
#9
01/27/2008 (4:57 pm)
You can use my resource for linked imagemaps, just make sure to add each imagemap to my system. It will create hidden static sprites for each imagemap, which should let the linked imagemap work fine.
#10
allowUnload is true, preload if false... I've tried everything. I've tried deleting the datablocks themselves, the Dynamic loader objects, I've tried clear, and remove... But the memory doesn't go down. Right now, seeing as my compiler won't compile the SDK... I kind of need this solution..
Thanks!
06/14/2009 (9:50 pm)
I've been trying to get this to work in my own program, and I've been watching task manager to watch the memory. It loads in things just fine (bringing the memory to a cheery 340,000 K - however, and this is what really needs to get done - the unloading from memory doesn't work. I'm completely perplexed as to why.allowUnload is true, preload if false... I've tried everything. I've tried deleting the datablocks themselves, the Dynamic loader objects, I've tried clear, and remove... But the memory doesn't go down. Right now, seeing as my compiler won't compile the SDK... I kind of need this solution..
Thanks!
Torque 3D Owner Rodney Rindels - Torqued
Divide By Zero