Loading a level crashes iTGB - Need Help Finding Crash Location - will pay who helps with programming services

#1 Okay... more searching around and digging into code and debug files revealed that this we are apparently choking the timer when iTGB loads on the iPhone, a timer that is disabled when running through XCode and Instruments.

So, now I am working on making the game load the levels in a different way - through a menu rather than when you start the game. After reading through numerous forum posts on level loading crashes, I'm about to ask the same question - my levels crash when I try and load them.

I have tried this 2 ways...

1.) Using a GUI screen with a button that when pressed, pops the GUI screen and loads the level.
2.) Using a separate 'MainMenu.t2d' level file with a button in it controlled by a behavior. You press it, that level closes and the new one loads. To load the level I am doing the following:

sceneWindow2D.schedule(40,endLevel);
sceneWindow2D.schedule(100, loadLevel, "game/data/levels/desertLevel.t2d" );

In the console.log, I can see that the level is fully loaded - and then we flat out crash. So... any ideas on how to address this?

#2 I would recommend that you attempt to use the proper tools instead of relying on the console output.
The console output will only give you iTGB engine dependent things which are unlikely to show up on the device just like that, especially as iTGB handles only very few of the common iPhone OS error messages in a way that exposes them to the log and scripting.

Its much more likely that it is either a memory or load time dependent issue.
The CPU as well as the Memory Inspektor will give you more information.
Also after the game crashes, the first thing I would recommend to do is checking the crash log. it will show you the exact C++ call stack that lead to the crash, giving you a possibility to identify the issue.

#3 Ok... I've determined it was crashing because I had objects with behaviors that used engine-level components to control their AI. The AI uses t2dVector a lot, and that seemed to make it very unhappy, so I added a script variable called $$GameEvents::gameLogicPaused which will cause the engine 'onUpdate' events to be skipped if it is true. So I now load the level with that as true - and now it loads without crashing! Only now the screen is black. Looks like the camera is in the right place, so I'm suspecting that the scenegraph is not getting properly updated... digging further!

#4 Ah! The scene is getting loaded, and the scenegraph is correct. The problem is that the datablocks, despite being loaded, cannot be found by the scene... that's odd, lol. Can't say I've seen that happen before...

#5 The biggest problem(s) I've had with level loading, which I don't have anymore at all due to a restructure of my code, are the following:

1. Behaviors don't always have their properties set before their onAddToScene method is called.
Resolution: I put doInit() calls in my onUpdate() to do my initialization.
2. Behavior order and object order obviously isn't guaranteed. I had no problems until one day I did something in the editor that changed the initialization order and all of a sudden everything started crashing.
Resolution: To fix this I reduced my behavior dependency and initialized all my behavior references in doInit() which is called from onUpdate(). I set an initialization flag to true once I've found all the behaviors the object needs and don't allow onUpdate processing to continue until I've found them all.

Schedule your endLevel and LoadLevel methods, I think your doing that already but thought I'd mention it. I put a 500 millisecond delay between my end and my next load for level transitions. I'll probably change that to daisy chain the end with the load but haven't gotten around to it.

Not Related to loads but I had problems with sqlite3 using up too much memory and crashing the program. So I had to download 3.6.16 and use the new sqlite3_soft_heap_limit(MAX_DATABASE_MEMORY); to limit it's memory to 1 megabyte (soft limit). Later found a prepared statement leak, but still think the limit is needed if your running close to memory and constantly loading objects from the database. My game is 95% database property driven so I do it a LOT.

Note: I should say now that I can load/unload levels for hours at a time without any problems after doing the above. One level barely fits into memory so I have to be really careful to free all my datablocks memory between level loads. I've shared this in the other thread about loading datablocks dynamically if you want the details.

#6 Aye, I'm seeing sorta the same thing here, now. I've gotten the level to load and all is fine - but as soon as I remove my 'gameLogicPaused' flag, the game instantly crashes. Watching the log, it's all kinds of things making it crash. The obvious initial one is always a call to 'getCurrentCameraArea' on SceneWindow2D. The odd thing is this crashes no matter how long I wait. I've loaded up the level, sat and waited for a solid minute, then dropped the console and unpause the logic - and bam - instant crash. So I'm certainly seeing things loading in varying orders, but I think there is also something funky going on with SceneWindow2D itself. The VERY weird thing, is I can call "sceneWindow2D.getCurrentCameraArea();" from the console without getting a crash. But call it from the engine - instant crash.

#7 I too would be very interested to find out what the hell is crashing. In the game I submitted to the App Store, I would have loved clearing out the entire level by reloading itself, but it was a guaranteed crash. I ended up having to clear out all the variables and reset all the objects manually.

Looks like you've gotten a lot further in investigating why it crashes - I wish I could be more useful, as this would solve a LOT of problems for me.

#8 Potentially check the resource unload function / texture unload function.
It at in earlier TGB versions was the reason for problems, for example dynamic unload did not work

#9 I'm really far along actually... I ended up writing a script-level 'load checker' that makes sure objects exists, however it still seems to be missing something. The game is still crashing, although it's a little bit random. However, the problem _ALWAYS_ is something with sceneWindow2D, but what crashes about it is totally random. Watching the XCode debugger, it always picks up something physics related on the object that sceneWindow2D is attached to that makes it crashes, or trying to get or set properties on sceneWindow2D itself which requires a t2dVector object creation. It's almost like sceneWindow2D is failing to get re-initialized properly after we manually load a level (the first level loaded in startGame never has any problems at all. In fact, even if we don't load a level in StartGame, and load a GUI instead, and load our level somewhere else, that new level still has sceneWindow2D issues - the only way not to have problems is to do all your level loading in startGame. Which means something about StartGame is properly starting up sceneWindow2D, while nothing else is starting it properly.

So the question is... what is it about working with a level loaded outside of startGame that causes sceneWindow2D to scream bloody murder? Solve that, and I think we've got this licked.

EDIT: To further prove this is the problem, I'm finding the following in my console.log:

playerCharacterComponent invalid object (sceneWindow2D)
playerCharacterComponent invalid object (sceneWindow2D)
playerCharacterComponent invalid object (sceneWindow2D)
playerCharacterComponent invalid object (sceneWindow2D)
playerCharacterComponent invalid object (sceneWindow2D)
playerCharacterComponent invalid object (sceneWindow2D)
collectableObjectComponent invalid object (sceneWindow2D)
collectableObjectComponent invalid object (sceneWindow2D)
collectableObjectComponent invalid object (sceneWindow2D)
collectableObjectComponent invalid object (sceneWindow2D)


Now that this occurs after sceneWindow2D has been created and is detected as being an object via isObject(sceneWindow2D);

#10 Ugh... finding more. This is a mess... I may need to rip a lot of this apart.

So first off, components are getting initialized before the scene has finished loading, which is causing problem. The main call that starts that off in the engine looks like it needs to be moved / rethought out.

After that is done, there is still a problem with sceneWindow2D that I have not yet determined...

The more I dig in, the more 'small things' I'm finding which cause crashes, which is why there are so many apparently random issues. There are a lot of ways loading can make this thing crash right now.

Digging back in... gonna be a long night....

#11 Some further progress. Game still randomly crashes, but it's becoming less random as I patch the holes. The big change I've made so far is I removed behavior adding when the object is created. I did this because behaviors often rely on other objects - and if the object does not exist that it relies on, we get problems. So now, all the objects are created, and after the entire world and all objects are loaded - THEN we add behaviors to everything. This removed numerous warning messages, and some of the random crashes.

But the game is still crashing... however it seems to be whittling down now to just an issue with SceneWindow2D where it is hating the fact I am trying to interact with it. Focusing on that, now...

EDIT: Pretty confused on why this is crashing. It crashes when setting a variable. Check it out:

//-----------------------------------------------------------------------------
// Set Gross Linear Velocity.
//-----------------------------------------------------------------------------
void t2dPhysics::setGrossLinearVelocity( const t2dVector& velocity )
{
    // Set Linear Velocity.
   mGrossLinearVelocity = 0.0f; //velocity;   //<------ CRASHES HERE!!!

    // Fetch Linear Magnitude.
    F32 linearMagnitude = mGrossLinearVelocity.len();

    // Clamp Linear Magnitude.
    if ( linearMagnitude > mMaxLinearVelocity )
        mGrossLinearVelocity.normalise( mMaxLinearVelocity );
    else if ( linearMagnitude < mMinLinearVelocity )
        mGrossLinearVelocity.zero();

    // Usage Flag.
    mGrossLinearVelocityActive = (mGrossLinearVelocity.len() > T2D_CONST_EPSILON);

    // Check for Zero-Clamp.
    if ( !mGrossLinearVelocityActive )
        mGrossLinearVelocity.zero();
}

That's a stock engine function. It crashes on the line I marked above. I changed that line to use 0.0f instead of the value passed in to rule out a bad value. Yet, VERY randomly, it crashes here. Anyone ever seen a crash like that before? Looking through the debugger information, I'm not seeing anything that would deduce what is going on there. Going to start digging into the physics code now and see where it's breaking, since clearly that variable getting changed from its previous is making the engine puke.

#12 Still pretty stumped on this current one. If anyone else feels like giving it a second pair of eyes, lemme know - I could really use some assistance on this one.

#13 Some more details on what I did that might help you:
1. I use to have the following behaviors:
PlayerComponent
MonsterComponent
AggressiveAIComponent
MovementComponent
LevelInformationComponent

All these behaviors use to collaborate and get added to objects as needed. However the interdependency of all these made initialization next to impossible. So switched over to:
PlayerComponent
MonsterComponent
LevelInformationComponent

And just made the old ones member objects that the objects create and use. This solved most of my interdependency problems. Now Player->LevelInformationComponent and Monster->LevelINformationComponent are the only dependencies, so both of them wait for the LevelInformationComponent to be loaded AND initialized before initializing themselves.

That was my first major change and was a big re-org of my code.

The second major changes was to NEVER EVER delete any of your scene objects in your destructors, not even using safeDelete(). Destructors are not safe to access or even look at objects that are in the scenegraph unless you know for a fact the scenegraph is still active and the object will be there. Because every object in the scene graph will be deleted by iTGB in random order when endLevel() is called.

I now put all my SceneObjects in the scenegraph and just use setVisible(true/false) to show/hide them during the level. I now never delete on endlevel but let iTGB handle all of that for me. I only delete objects that are temporarily created. This might seem like a performance dog but I frequently have 300-400 objects in the scenegraph without problem, though their not all visible at one time obviously.


The deletion problem was the hardest for me to get right because as a C++ developer it's second nature for me to clean up after myself in destructors, and you have to be very very careful what you clean up in destructors with iTGB.

#14 After banging away at this for a week now, I've just about got it where loading works fine. I fixed a few bugs in the engine itself, modified my own code, and have come out with a much improved way of doing all this.

I'm now down to one totally random crash. Random most likely means 'something' is not loading in order. What is happening is that I have a function that is meant to be called only after all objects are loaded. This function will activate the behaviors on all the objects and get them setup. Part of this function is to call 'onAddToScene'. Most of the time, this works fine. Then, occasionally, the game will crash after loading a level, and give me the following error for every object in the scene:

common/gameScripts/levelManagement.cs (449): Unknown command onAddToScene.
  Object (1374) t2dStaticSprite -> t2dSceneObject -> BehaviorComponent -> DynamicConsoleMethodComponent -> SimComponent -> SimObject

Since this affects EVERY object in the scene, I'd say it's safe to say that something pretty low-level is not always getting initialized properly when a new level is loaded. I'm finding definitions for onAddToScene throughout the code tree, however it only goes as high as 'SimComponent'. That makes me wonder... has SimComponent not loaded properly? That seems a bit weird, since SimComponent works fine in the first level loaded. But then it flat out randomly chokes when we load a second level.

I'm gonna go sleep on this... but I'm posting to see if anyone might have some insight to SimComponent in terms of it loading/unloading when we load/unload levels? Anything that might cause it to not load properly (at what appears to be in a random manner)? Anything that I might be able to check to insure it is loaded before having my code continue a load?

#15 Jeez, this thread is just over the top for me. If you get this fixed and we can all start loading levels at will without having to cross our fingers, pray to God, or recite a magic incantation, I'll seriously toss you some money.

#16 @Raphael
I've got mine working. It's been working for months. I actually support teleportation between levels (mark/recall potions) and teleport frequently during gameplay. These issues are not problems with iTGB, but problems in native behaviors we develop. native behaviors are easy to develop until you add in dependencies between objects, then it gets really tricky.

I've tried to outline the issues in my posts that you have to address when using behaviors that are dependent upon other behaviors. It's very tricky and the slightest flaw in your logic will cause these random crashes.

I've run my game for hours, teleporting between levels frequently to buy/sell etc without any crashes.


@Dave
I'm not sure if there is a reliable way to determine if all objects have finished loading. Essentially what I did was have each object locate the object they need and in their onUpdate method activate itself after their all of it's dependent objects are initialized.

So in my OnUpdate method for all my behaviors the first thing is:

if (mInitialized == false) {
 doInit();
}
if (mInitialized == false)
  return;

.... update code.

In the doInit I only set mInitialized after I've:
1. Located all the dependent objects I need.
2. All the dependent objects are initialized().

It's tricky though because if the dependency is bi-directional then you have a deadlock scenario and you need to use 2 phase initialization. I removed all my bi-directional dependencies because of this. I have 1 master behavior called LevelInformation that depends on nothing. All the other behaviors depend on it and wait for it to be initialized before initializing themselves.

My other posts tried to explain it at high level because more than likely the details of what I did aren't going to help much because it's very implementation dependent.



Another thing I did was to have objects register themselves with the global LevelInformation object when their ready for interaction with others.

LevelInformation - Singleton that everyone needs and nothign happens in the game until this behavior is created and fully initialized.
MonsterComponent - Component of enemies the player can fight. This object needs LevelInformation only. Registers itself with LevelInformation once it's initialized.
PlayerComponent - You the player. This object needs LevelInformation only. Registers itself with LevelInformation once it's initialized.

Players/Monsters interact with each other only through the level information component. Player asks the Level for the list of monsters, and the monster AI component asks the level for the local player(s).

By using this registration mechanism I minimized the bi-directional dependency initialization complexity between player/monster/ai/movement etc components.

I use to have a ton of components, now I have 3 components only and the old components are now just regular worker objects that are created by the behaviors.

#17 Some of this feels like work around's to a problem in the behavior system. The system is built and meant to allow interaction and dependency between objects - if the only problem is that the objects need a guarantee that other objects exist, then it seems like that's really where the focus should be in the code.

The 'doInit' solution sounds like something that should get built into the engine. Behavior components already use something like this, although it's not that strong since it just makes sure an object exists, which isn't enough of a check. Perhaps expanding things to an additional behavior function that is called at the start of every engine-level 'onUpdate' that checks if there are object dependencies, then checks for their init state before allowing the behavior to run. This, coupled with the engine changes I've already made that makes it so behaviors do not get attached to objects until everything has been loaded seems like it MIGHT work.... digging in and trying.

(I'm also trying to make a solution that can be rolled back into the engine so users don't keep having this problem. There are SO many people bitching about this on the forums, it's gotta be causing GG some headaches...)

EDIT: After I stepped away and thought about this some more, I began thinking about the debug output I was getting. It shows that every single object in the scene is saying it does not have certain basic SimComponent properties available yet. While the doInit check would get around this, I think the problem is actually deeper, and it has something to do with a very low level object not getting loaded before other things... the doInit check relies on that getting loaded, so it works. But it's still a work around... the problem is deeper than this. Somewhere between t2dSceneObject and SimComponent....

#18 I posted something on this a while back when I first ran into the situation. To fix these problems we would need the following to be guaranteed to occur in order.
1. All Scene Objects are created.
2. All behavior are created.
3. All editor assigned properties are set on the behavior
4. all behaviors are added to their t2dSceneObjects.
5. All behaviors onComponentAdd methods are called.
6. All the objects are then added to the scene in one burst.

#19 @Bret: The changes I've been doing actually go through a bunch of that. What I'm implementing literally as I'm typing this is the ability for an object to know if its behavior is active or not. Then, in the beginning of the onUpdate for a component, for it to ask any associated objects if their components / behaviors have been properly initialized - and if they have not, we return on the spot.

I'm getting closer every day to getting this patched. Hopefully I'll have it dealt with very soon (since it's holding me back on a contract gig that's coming close to the end). Once I get this resolved I'll do another resource and see if I can get it added into head.

#20 Small update....

I've discovered that when our crashes occur, it's because none - not even one - of the components or behaviors have been properly loaded. I've gotten the game to a point where it runs numerous checks, which now prevents it from crashing. But where the game would have crashed before, it instead gives and error on every component and behavior I'm checking, and says none of them exist at all.

Not there yet, but getting closer. Now to determine what that weird circumstance is that prevents all our behaviors/components from getting fully loaded.... (I say fully because they do exist to an extent - but its like they were started to be loaded then just stopped so they don't 100% work...)