Game Development Community

T3D 1.1 Preview - Many Object/Ghosts Causes FUBAR with Ai - LOGGED (THREED-1780)

by Steve Acaster · in Torque 3D Professional · 05/02/2011 (11:13 am) · 12 replies

T3D 1.1 Preview

Platform:
win7 32bit

Target:
world editor -> object count -> Ai -> Player_UpdatePos

Issue:
I'm under the impression that the 1024 max object count (or was it 4096 ghosts?) has been fixed in Preview *(edit: it hasn't but that's not what is involved here anyway)*, but I've found an issue which causes lockup when introducing Ai.

[edit_ignore]In Beta3 a community member posted a fix for the limit (linky) and all was well.[/edit_ignore]

With over 3000 objects (mostly markers) everything slows and dies when Ai are introduced but are fine until Ai spawn, and it's the Ai constantly looping Player_UpdatePos that seems to be the issue - so me thinks ghosting issue.


Repeat:
You need a mission with a lot of objects and some Ai.
torque.abigholeintheweb.com/public_system/useruploads/Yorks_Object_Lag.zip
Thanks to E.B. for hosting.

Place the mission file and AIplayer.cs (it's got some easy functions to help) in the appropriate places and boot up the mission. In console get your FPS on screen:
metrics(fps);
then call:
bigspawn()

- and 32 Ai will appear from the functions included in modded aiplayer.cs. Now watch your FPS die and bottom out at 0.6fps or less. Run a Profile (CTRL F3 - and hold it for a load of seconds). Then check the profiler and see that Player_updatePos is going bonkers.
Profiler Data Dump:
Ordered by non-sub total time -
%%NSTime  %% Time  Invoke #  Name
 91.871  92.087    14084 Player_UpdatePos// <--- BONKERS!
  3.097   3.097  1351588 ShapeBase_ProcessTick
  1.127  99.358      448 ProcessList_AdvanceObjects
  1.059  95.799   677152 StdServerProcessList_OnTickObject
  1.023   2.432   677152 StdClientProcessList_OnTickObject

1.1 Beta 3
0.127   0.405    15429 Player_UpdatePos

Suggest:
Fix it!

#1
05/02/2011 (11:38 am)
Logged as THREED-1780.
#2
05/02/2011 (11:59 am)
Update:
Actually the fix wasn't in like I thought it was --- but even so, keeping ghosts/objects under 4096 in netConnection.h results in this odd Player_UpdatePos regardless of whether the object limit it raised or not ... so ignore my ramblings about a fix, there's still something odd happening with object counts and introducing Ai.
#3
05/16/2011 (6:54 pm)
I had wondered if it was just a thing with markerObjectType but using find-in-files to replace the markerObjects with the standard TsStatic cube (no collision)causes the same issue, introduce any sizeable number of ai/players and performance grinds to a halt and "Player_UpdatePos" goes through the roof.
#4
10/20/2011 (6:34 am)
Okay so I found the culprit here, and it's pretty crazy. The bug is in the new Mission Area editor they introduced (re-introduced?) in 1.1F. And it's not really a bug, per se, but just a really really poor design choice.

in Player:updatePos, they call Player::checkMissionArea, presumably to allow the MissionArea system to check if the player's in bounds, etc.

setPosition( newPos, mRot );
   setMaskBits( MoveMask );
   updateContainer();

   if (!isGhost())  
   {
      // Collisions are only queued on the server and can be
      // generated by either updateMove or updatePos
      notifyCollision();

      // Do mission area callbacks on the server as well
      checkMissionArea();
   }

The first thing checkMissionArea does call MissionArea::getServerObject, which is just about the worst function ever. Because what MissionArea::getServerObject does is the most gawd-awful brute force search of iterate over every ghost-always object in the entire level looking for the MissionArea object. And then it does a dynamic cast inside the loop, to figure out if it's the Mission Area object - which down inside the RTTI library, results in a string compare against types.

MissionArea * MissionArea::getServerObject()
{
   SimSet * scopeAlwaysSet = Sim::getGhostAlwaysSet();
   for(SimSet::iterator itr = scopeAlwaysSet->begin(); itr != scopeAlwaysSet->end(); itr++)
   {
      MissionArea * ma = dynamic_cast<MissionArea*>(*itr);
      if(ma)
      {
         AssertFatal(ma->isServerObject(), "MissionArea::getServerObject: found client object in ghost always set!");
         return(ma);
      }
   }
   return(0);
}

I'm not 100% sure, but I'm fairly certain the top of the list of things to not do inside the update tick for every player in the world you'll find iterate over every object inside the world.

Anyway, the quick hacky fix for me was just to remove the call to CheckMissionArea in Player::updatePos. I don't use the MissionArea object, as it didn't even exist prior to this update, and it certainly doesn't exist in any of my levels - most of which contain upwards of five thousand objects in them.

Their are a number of ways for the GarageGames guys to fix this the right way: Do the search once, at the beginning of time, and squirrel away the pointer, as doesn't change throughout the level's life time; Give the MissionArea ojbect a designated name, like we do for LevelInfo, and search for it using the string hash search, which is pretty fast, that is already in place in findObject; or any one of a number of other solutions.

But right now I'm looking to see if there are any other places in the code that might be calling CheckMissionArea, and making sure those are excised as well.

Good coding!
#5
10/20/2011 (10:14 am)
OOH, good catch!
Flying vehicle uses MissionArea::getServerObject(), and I've been seeing some hideous performance issues with multiple AI pilots that I just assumed was due to the rigid body stuff. This could well be the issue. It seems like such an innocuous function until you dive into it!

Looking back to TGEA, this used to be achieved using
MissionArea * obj = dynamic_cast<MissionArea*>(Sim::findObject("GlobalMissionArea"));
Which, as you say, is not so nasty.
#6
10/20/2011 (11:14 am)
Memory is kind of vague as to when the changes were made, but could this be the result of the fix that obviated the need for a specific name of the MissionArea in order for it's callbacks to work? Which was an issue introduced in TGEa...
#7
10/20/2011 (11:26 pm)
If I remember correctly the editor for the beta versions of T3D 1.0 removed support for seeing the MissionArea, then the MissionArea was mostly removed then added back in sometime.

Not having to have a specific name for the MissionArea would only be good if you wanted to have more than one in each level (which the current editor doesn't allow)

This may also be of use to see the area again:

http://www.garagegames.com/community/blogs/view/20793

#8
10/21/2011 (4:47 am)
Nicely spotted, Dusty. :)
#10
11/13/2012 (2:38 pm)
Was that not fixed in MIT? Really?
#11
11/13/2012 (2:46 pm)
Well spotted Daniel, this still hasn't been fixed.
#12
11/14/2012 (8:55 am)
What hasn't been found? <duck and run>