Preload Materials

#1 i dont know if it changes with preload materials is set to true but i am very sure without preloading materials, torque loads every material in txb, i want to prevent this but i dont know if there is a way.

because my game has 10 levels and the levels are completely different from each other, and at the runtime my game uses approximately 1.3 GB RAM after 3-5 hours continuous playing it throws memoryOutOfRange exception. :S

Serkan.

#2 This scares me. I'd love a GG post on this. I need to preload to avoid hitching but if torque is loading every material in the builder my game wont run once I get the other levels in.

:(

#3 This is the way it currently works. The simple solution is to write a little program that parses the .txscene files (your levels) and throw away all unused materials.

#4 Thanks

Not a deal breaker but sad. :(

I can't believe the builder was designed like that. Oh well. :)

out of curiosity. (save me the testing if someone knows ;) )

I load 2 .txscene files one with shared objects and one for the level.

Mats in both scene files will only load once correct? Or will a single mat in both .txscenes load twice as unique materials?

Also I'm hoping that the shared materials don't have to exist in both .txscenes and if they do I hope torque is smart enough to only load them once.

I'll implement these changes soon. If anyone knows exactly how torque handles the things I mention above I'd love to know.

Thanks!

#5 There are a few intermingled points of interest here. And they are quite important if you are looking at either memory usage or preloading - and if you are looking at both then they are doubly important.


Firstly, Christian is correct. TX currently works by loading everything that a scene file references. And that often means everything in your game, because txb saves references for ALL materials when it saves a scene, even if the scene's objects don't reference the materials.

Secondly, TX will create NEW instances of all your materials every time you load a level. This means that (1) it won't take advantage of a previous level/scene having already used a material; it will load it again, and (2) if you have more than one scene loaded at the same time then you will have MULTIPLE copies of your materials in memory (e.g. transitions between scenes can require both scenes being loaded at the same time - I certainly use multiple loaded scenes myself as it's very useful).

Thirdly, the 'preload materials' setting does not modify whether or not materials are read from disk into memory. It actually refers to setting up loaded materials so they are ready to use - the texture assets themselves have already been loaded by this point.


So what to do? Well as christian pointed out you can run a script on the scene files to strip out any references to materials not used by objects in the scene. You could set this as a task for VS to run when you compile your game. This might be enough for you if your levels aren't needing too many assets, but were referencing loads they didn't need.

But that won't solve the issue of mulitple instances of materials if you have multiple scenes loaded. It is, of course, time to get modding the TX engine :) The simplest thing to do that gives you the biggest bang for your buck, is to mod TX so that it uses cached textures (like it really ought to be doing anyway). Go into the TorqueSceneData class and comment out the bits where it creates a new ContentManager when a level is loaded - and also comment out several other places where it pushes/pops/disposes content manager instances.

Then in you Game - before you load any levels - create a new XNA ContentManager instance and push it onto the ResourceManager's stack of content managers. That's all - now textures that are loaded into memory once will stay in memory and the next time you load a scene that needs texture X it won't have to load it from disk. And of course, you won't ever get any duplicate instances of your materials so if you have multiple scenes loaded at one time they will use a lot less memory.


If you need to go further than this and load/unload content dynamically then you will need to load content into different content managers. If that's your bag, then you'll probably want to create your own custom framework for handling the details of what gets loaded/unload when (the needs of your game will dictate how complex this framework needs to be).

#6 Thanks for the Info Duncan it is really helpfully :).

#7 Thanks for the reply, but i am trying to understand what do you mean by

Quote:"You could set this as a task for VS to run when you compile your game. This might be enough for you if your levels aren't needing too many assets, but were referencing loads they didn't need."

I really need to solve this issue of mine, this seems the only way for me but i havent got slightest idea how to do it, if you can you point me a direction i will be very glad.

thanks again

Serkan.

#8 Hi Serkan I'm not going to be able to get to it for a bit but when I do I'll be happy to share my code/program with you. I'm going to mod the engine as Duncan suggests and then write a program that goes through and rips out material references from the .tx file. The program will just read from a list of materials that I would like to remove.

If you don't have time to wait you'll need to write a program that rips out the mat references and then tell VS to run the program (not sure how to make VS do that I have to learn myself) :)

#9 Thanks Mat ;) for your help,

while i was waiting an answer i tried to remove unnecessary entries manually from my level01.txscene file, after that i build my game and run, gues what? it worked :D

before that when my game loaded level01, it was using approximately 1.2 GB of Ram but now its using 200 MB, after i finished that level, i loaded another unchanged one and as expected it loads all the stuff, it used 1.2 GB. I played 3-4 more levels and there weren't any error.

so even if i couldn't find a way to do it systematically or with code, i can do it manually, its my insurance :D of course there is a drawback doing it manually, such as, after opening builder and open that level and saving it, even if you didn't change anything, it overrides the file with its defaults, that is; it adds whole references again.

luckily while i was creating my levels, i used different prefixes for different level images, for example all the images i used on level01 starts with a, such as "aBalloon01.png" and "aSharkDieAnim.png" etc.
i think this will help me during my parsing attempt of the .txscene files.

Thanks again mate.

SrkN.

#10 Glad things or working for you. I'll let you know once I have made a tool that does it if you or anyone in the community has use for it :).

#11 @Matt: you could make it more automated by having your program identifying what materials a scene's objects reference by looking at all the <Material nameRef="mymaterial" /> entries and also materials used by animated sprites via backtracking the <AnimationData nameRef="myanimdata" /> entries. Hopefully it shouldn't be too bad to do that as it's in xml so you should end up with lists of anims, materials, etc, when you're program reads in the scene file anyway.

#12 Duncan thanks to you too, you helped us so much,

Matt, i am trying to write it too, but i am not very experienced in c#, so while i am writing it, i will check weather you finished yours or not :D.

Thanks for sharing.

SrkN.

#13 First of thanks again Duncan. That will help a ton.


Here is the engine code modification as Duncan mentioned go into the TorqueSceneData class and comment out these lines of code.

if (_contentManager == null)
{
contentManager = new ContentManager(TorqueEngineComponent.Instance.Game.Services);
_contentManager.RootDirectory = ResourceManager.Instance.CurrentContentManager.RootDirectory;
                ResourceManager.Instance.PushContentManager(_contentManager);
}

and

// drop the content manager
_contentManager.Unload();
_contentManager.Dispose();
_contentManager = null;

Add the first bit of code right in front of your level load and the 2nd bit when you unload.

It will be a few days before I get to writing the other part. I need to stay ahead of my animator and have some other coding to do :). I'll post it when it's done though.

Thanks again Duncan

Gl Serkan.

#14

Quote:Add the first bit of code right in front of your level load and the 2nd bit when you unload.

Hopefully you aren't adding those bits of code when loading/unloading a level. Or that would be the same situation as TX was doing before :)


Btw, you also need to comment out: ResourceManager.Instance.PopContentManager();


Then when your game starts up you create a content manager yourself and do a ResourceManager.Instance.PushContentManager(myContentManager);

Then, assuming you have simple content management requirements, just leave that content manager in place for the rest of the game - textures will now be cached. NOTE: for unloading content you can call Unload() on myContentManager. Serkan might want to do this as he clearly needs to unload one level and load the next one.

Also note that you can have multiple content managers though. So for example, you can have content manager A which has content you always want loaded (maybe stuff that the UI/HUD uses, for example), and then content manager B that you use to load/unload content for each level (that is, you can load/unload blocks of content by using a different content manager for each block of content). As you can see you can get pretty complex with content management if you need to or alternatively keep it simple if you don't need more advanced content management.

At the very least though, you probably want to have a single content manager that exists for the duration of the game so that loaded textures are reused instead of being thrown away and reloaded everytime you load/unload a scene.

#15 Awesome thanks for the feedback. I am only creating a content manager once but missed the ResourceManager.Instance.PopContentManager(); line :) I'll need to comment that out.

Also I'll need to nuke these lines
# _contentManager.Dispose();
# _contentManager = null;

not sure what I was thinking (I hadn't tested reloading a level after completing one and was just exiting but am sure I would have crashed lol.)


Thanks again!

#16 Thanks boys, i am working on it, i will let you know as soon as i finished my testings.

SrkN.

#17 Heya Duncan. One more question for you. Everything works great except for one thing.

If I call _contentManager.Unload when unloading a level and then reload the level I get an error in the EffectManager.cs "Cannon access a disposed object. Object name: 'Texture2D'.

if I simply don't call _contentManager.Unload or just nuke the content manager and recreate it when loading a new set of .txfiles everything works.

Below is the code causing the error:

in my game.cs right after begin run:

if (_contentManager == null)
            {
                _contentManager = new Microsoft.Xna.Framework.Content.ContentManager(GarageGames.Torque.XNA.TorqueEngineComponent.Instance.Game.Services);
                _contentManager.RootDirectory = ResourceManager.Instance.CurrentContentManager.RootDirectory;
                ResourceManager.Instance.PushContentManager(_contentManager);
            }

Then In my LoadingControl.cs I load my level and character data

ame.Instance.SceneLoader.OnSceneLoaded = null;
                    //alweays load shared data first.
                    Game.Instance.SceneLoader.Load(@"data\levels\characterData.txscene");
                    T2DSceneGraph _sceneGraph = (T2DSceneGraph)TorqueObjectDatabase.Instance.FindObject<T2DSceneGraph>("DefaultSceneGraph");
                    _sceneGraph.Name = "LevelSceneGraph";

                    Game.Instance.SceneLoader.OnSceneLoaded = SceneLoaded;
                    Game.Instance.SceneLoader.Load(@"data\levels\trible.txscene");

Then later I unload the level:

//find all objects and nuke em as torque apparently misses some as I've
//read on the forums
List<ISceneContainerObject> _allObjects = new List<ISceneContainerObject>();  
                    T2DSceneGraph _sceneGraph = (T2DSceneGraph)TorqueObjectDatabase.Instance.FindObject<T2DSceneGraph>("DefaultSceneGraph");  
                    Vector2 _vector = new Vector2();  
                    _sceneGraph.FindObjects(_vector, 9999, TorqueObjectType.AllObjects, (uint)0xFFFFFFFF, _allObjects);  
                    foreach (T2DSceneObject _obj in _allObjects)  
                    {  
                        if (_obj is T2DSceneObject)  
                        {  
                           _obj.MarkForDelete = true;  
                        }  
                    }
                    //Unload Character scene
                    List<ISceneContainerObject> _allCharObjects = new List<ISceneContainerObject>();
                    _sceneGraph = (T2DSceneGraph)TorqueObjectDatabase.Instance.FindObject<T2DSceneGraph>("LevelSceneGraph");
                    Vector2 _vectorch = new Vector2();
                    _sceneGraph.FindObjects(_vectorch, 9999, TorqueObjectType.AllObjects, (uint)0xFFFFFFFF, _allCharObjects);
                    foreach (T2DSceneObject _obj in _allCharObjects)
                    {
                        if (_obj is T2DSceneObject)
                        {
                            _obj.MarkForDelete = true;
                        }
                    }

//stopLevelSound
                    sound.loading.SoundLoading.Instance.EndLevelSound();
                    //unload
                    //unload level data
                    Game.Instance.SceneLoader.UnloadLastScene();
                    //unload character data
                    Game.Instance.SceneLoader.Unload(@"data\levels\characterData.txscene");                   
                    GC.Collect();
                    GC.WaitForPendingFinalizers();
                    _currentMenu = MenuSelect.winning;
                    guiWinning = new GuiWinScreen();
                    GUICanvas.Instance.SetContentControl(guiWinning);
_contentManager.Unload();

After that In my LoadingControl.cs I call the same code above to load my level and character data.

What should I be doing different with the content manager? I'm assuming I need to call unload to remove textures from memory.

#18 hmmm, you should be able to unload assets from a content manager and then load new ones into the same content manager. However, there's no harm in using a new content manager for each level.


Just a thought: does the crash happen when you *don't* try to reload the level?

#19 Your level unloading does seem a little convoluted though. Is there a reason why you are unloading your level objects by traversing the scenegraph, but unloading the characterData by calling Unload?

#20 No the crash only happens when I reload the same level.

The unloading does look crazy.
So I am actually calling unload on both the char data and the level data the UnloadLastScene() is calling unload on the level data (this way I don't have to call the name of my level file but the name of my character data is alweays the same) I may eventually never unload the char data.

//unload level data  
                     Game.Instance.SceneLoader.UnloadLastScene();  
                     //unload character data  
                     Game.Instance.SceneLoader.Unload(@"data\levels

The rest of the crazy code is due too a bug in the .Unload method. I was having issues before where torque didn't actually unload the .txscene files.

I found a few forum posts on the issue and they recommended manually deleting everything in the scene and then the scene graph.

Here is one of those posts
http://www.garagegames.com/community/forums/viewthread/76056

I just wanted to run the code by you to make sure I wasn't doing something stupid.

If creating a new content manager for a new level sounds like an ok solution to you I'll do that :).

Thanks again.