AI kits and game performance...
by Paul Weston · in Torque 3D Professional · 09/15/2011 (8:37 am) · 24 replies
Hi all,
As our project nears completion and the game environment gets larger and more complex, we are noticing a real performance hit when we have the AI in the game. Even a small group of under 10 AI characters seems to really bog things down.
We are using the Universal AI Starter Kit, which we love for its simplicity and easy integration, but I can't help but wonder if perhaps our particular environment may be better suited to a different AI solution.
Wondering if anyone has any experience with the Tactical AI Kit - the demo works very well and has dozens of AI running in all directions and responding very quickly without any noticeable performance issues. Granted, the gameplay area in this demo may be markedly smaller and less complex than the entire interior of the Starship Enterprise, which is where we want our AI to be, but still the performance seems so good I am wondering if they have found a better way of handling the memory, etc, than we currently have with the UAISK.
No slight against Twisted Genius, his kit has more than served its purpose for us during the development of our project, but if there is a faster and less-intensive way to handle AI so we can have 10-20 characters on the ship without it slowing to a crawl, I'd be interested...
Basically, beaming out the AI crew increases framerate dramatically, even if only 3 out of the 10 default crew members is walking on a path and the others are stationary. Seems that the engine should be able to handle AI better than this?
Anyone have any experience in this area?
Many thanks!
As our project nears completion and the game environment gets larger and more complex, we are noticing a real performance hit when we have the AI in the game. Even a small group of under 10 AI characters seems to really bog things down.
We are using the Universal AI Starter Kit, which we love for its simplicity and easy integration, but I can't help but wonder if perhaps our particular environment may be better suited to a different AI solution.
Wondering if anyone has any experience with the Tactical AI Kit - the demo works very well and has dozens of AI running in all directions and responding very quickly without any noticeable performance issues. Granted, the gameplay area in this demo may be markedly smaller and less complex than the entire interior of the Starship Enterprise, which is where we want our AI to be, but still the performance seems so good I am wondering if they have found a better way of handling the memory, etc, than we currently have with the UAISK.
No slight against Twisted Genius, his kit has more than served its purpose for us during the development of our project, but if there is a faster and less-intensive way to handle AI so we can have 10-20 characters on the ship without it slowing to a crawl, I'd be interested...
Basically, beaming out the AI crew increases framerate dramatically, even if only 3 out of the 10 default crew members is walking on a path and the others are stationary. Seems that the engine should be able to handle AI better than this?
Anyone have any experience in this area?
Many thanks!
About the author
Programming team lead, Gambit Realm development studio. Based in Ottawa, ON, Canada.
#2
09/15/2011 (9:16 am)
And TJ might have some insight on this as well. Drop him a line and see if he can point to some way to streamline the "think" methods, perhaps based on what the particular AI unit needs to do. Then a few datablock entries can be used to toggle features in the AI kit....
#3
Could it be that the static AIPlayers are somehow conflicting with the AISK-spawned ones? We would have just put them in as static objects and not even AIPlayers, but with AIPlayer we get a nice head movement that makes them look more alive, so that's why we went with the static AIPlayer solution for the sitting characters.
As far as I can see, we are using the kit exactly as intended, with the various datablocks, etc, configured depending on what each character's purpose is.
As for framerates, I don't have specific numbers, just can really tell the difference the minute I remove the AI. With the entire ship loaded, one can actually navigate it as long as there is no AI. The minute we add the crew, everything gets pretty choppy and it's hard to walk around because of the delay when you turn, etc. AI crew does not impact performance so much in our demo, but that only includes the bridge and the 3 decks around the shuttlebay, not the other 20-odd decks in the ship.
Could be though that this is actually related to our character model, perhaps? We had to customize it / blow it up a bit because its size did not match our scale of the ship. Can't remember which character we purchased, there were a few before we settled on what we have now and customized it. I'll ask Brian, he's the artist and worked on all that. Maybe I should try using a stock model like Gideon or Kork and see if performance is better.
Encouraging that you are able to have 30 AI without issue, so maybe this is just a combination of bad design on our part as it relates to character models and mixing of static AI and AISK-spawned AI...
Anyhow, any more thoughts, please keep them coming :)
09/15/2011 (9:43 am)
Well, our default AI set consists of a bridge crew and 2 guys in the shuttlebay. Bridge crew includes 2 guys walking on paths, spawned via markers and using the AISK, and 4 static characters which we just add in as new AIPlayer in the mission file (they are all seated so have a special sitting animation, they are unarmed and don't have to think or anything so we just created them statically). Shuttlebay crew is two AISK-spawned characters, one on a path and the other standing in one place, occasionally pacing.Could it be that the static AIPlayers are somehow conflicting with the AISK-spawned ones? We would have just put them in as static objects and not even AIPlayers, but with AIPlayer we get a nice head movement that makes them look more alive, so that's why we went with the static AIPlayer solution for the sitting characters.
As far as I can see, we are using the kit exactly as intended, with the various datablocks, etc, configured depending on what each character's purpose is.
As for framerates, I don't have specific numbers, just can really tell the difference the minute I remove the AI. With the entire ship loaded, one can actually navigate it as long as there is no AI. The minute we add the crew, everything gets pretty choppy and it's hard to walk around because of the delay when you turn, etc. AI crew does not impact performance so much in our demo, but that only includes the bridge and the 3 decks around the shuttlebay, not the other 20-odd decks in the ship.
Could be though that this is actually related to our character model, perhaps? We had to customize it / blow it up a bit because its size did not match our scale of the ship. Can't remember which character we purchased, there were a few before we settled on what we have now and customized it. I'll ask Brian, he's the artist and worked on all that. Maybe I should try using a stock model like Gideon or Kork and see if performance is better.
Encouraging that you are able to have 30 AI without issue, so maybe this is just a combination of bad design on our part as it relates to character models and mixing of static AI and AISK-spawned AI...
Anyhow, any more thoughts, please keep them coming :)
#4
If it turns out to be the models, you definitely want to take advantage of zones and culling.
09/15/2011 (10:35 am)
You could also "turn off" AI that is out of range of anyone - basically, if I can't see 'em they can shut down. Fire them back up when someone is near. Perhaps use triggers around their areas or see if zones have some functionality that you could use to help.If it turns out to be the models, you definitely want to take advantage of zones and culling.
#5
09/15/2011 (11:07 am)
TJ would have a better insight into how UAISK works, so may be able to pin-point a few things for you to try. Try a few things like remove the seated AI players, there may be a conflict. Are you using UAISK for them, or a second basic AI script ? it may be that the whole process is conflicting, getting confused and doing more work than it should. Plus try another stock model, I take it your bone structure is setup correctly?
#6
edit: then removing a different floors AI, or reset them to do nothing other than stay static, and then kick in the AI routines when needed.
09/15/2011 (11:09 am)
Another thought, is that if this is a single player game, maybe you don't need to spawn additional AI in the game until a trigger is met, reducing the numbers a bit. If you have multiple floors with AI on.edit: then removing a different floors AI, or reset them to do nothing other than stay static, and then kick in the AI routines when needed.
#7
Getting down to business, we need to determine where the issue is.
File: aiSpawning.cs
Function: AIPlayer::spawn()
At the bottom of the function you'll see a pair if/else statements that have schedules in them. Completely comment out them both. This will turn off the AI's thinking process after spawning. After you comment that code out, if you're still having performance problems, then it's due to either the art your using or just the stock "AIPlayer" class in general (as The UAISK has been effectively turned off after spawning).
But no matter the results of the above test, as Richard Ranft and S2P both suggested, I'd also recommend trigger based spawning/despawning to only have the AI you really need be active at any given time. This topic also has a bit more information on both these subjects. Feel free to ask me anything if you require additional assistance.
EDIT: If your sitting NPCs use a separate AI system that has a "think" cycle, do a separate set of tests in which you turn that off; as the issue could be in that AI system.
09/15/2011 (11:55 am)
First off, in general, AI is considered a fairly processor intensive task. You're bound to see some type of a performance hit with AI spawned versus not spawned, no matter which AI solution you go with. Some "AAA" games even allot AI a CPU budget, much like they do for physics and rending, due to how performance intensive it can be. I'm not sure what your framerate is without AI (since you didn't give exact numbers); but if it was barely holding on before, then adding AI in would be enough to push it over the edge.Getting down to business, we need to determine where the issue is.
File: aiSpawning.cs
Function: AIPlayer::spawn()
At the bottom of the function you'll see a pair if/else statements that have schedules in them. Completely comment out them both. This will turn off the AI's thinking process after spawning. After you comment that code out, if you're still having performance problems, then it's due to either the art your using or just the stock "AIPlayer" class in general (as The UAISK has been effectively turned off after spawning).
But no matter the results of the above test, as Richard Ranft and S2P both suggested, I'd also recommend trigger based spawning/despawning to only have the AI you really need be active at any given time. This topic also has a bit more information on both these subjects. Feel free to ask me anything if you require additional assistance.
EDIT: If your sitting NPCs use a separate AI system that has a "think" cycle, do a separate set of tests in which you turn that off; as the issue could be in that AI system.
#8
Thought of the trigger-based solution, however doesn't really suit things most of the time - if for example I am hunting Klingons in the lower decks (one mission has a bunch of them running around you have to kill), I don't want them to despawn/respawn if you get too far away and/or decide to go work on a different enemy somewhere else if one is beating you up too badly - if we despawn and respawn, the one that the player was fighting and could have inflicted half damage on, would respawn at full health if this was set up. Kind of unfair to the player who may have just wanted to go up a couple of decks to the armory to replenish, etc.
Anyhow, all good thoughts, as I said will try some out over the weekend.
Cheers,
P
09/16/2011 (6:36 am)
Thanks TJ and everyone for the advice... Will be working on things over the weekend and see if I can make any progress.Thought of the trigger-based solution, however doesn't really suit things most of the time - if for example I am hunting Klingons in the lower decks (one mission has a bunch of them running around you have to kill), I don't want them to despawn/respawn if you get too far away and/or decide to go work on a different enemy somewhere else if one is beating you up too badly - if we despawn and respawn, the one that the player was fighting and could have inflicted half damage on, would respawn at full health if this was set up. Kind of unfair to the player who may have just wanted to go up a couple of decks to the armory to replenish, etc.
Anyhow, all good thoughts, as I said will try some out over the weekend.
Cheers,
P
#9
09/16/2011 (7:03 am)
That's not so - you can always hold their state info and restore then when you respawn them. You can also store their locations and put them back, or have them spawn elsewhere if they've been encountered ('cause if I was hurt, I'd move if I had the chance).
#10
09/16/2011 (7:42 am)
Hmm, neat idea... Is there a stock way of doing that, saving the AI characters' states I mean? Or is it something I would have to manually code that would basically write the various state items (location, health, etc) to variables when despawning a character, and then recall them at the next respawn?
#11
Also splitting up the timings of Ai think routine can help, rather than having them all called at the same time.
09/16/2011 (7:49 am)
Remember, skinned-meshes (Ai, Players, etc) are rendered via the CPU and not the GPU. T3D is not multithreaded as stock. Every instance of an Ai/Player, as well as every bone and poly adds stress to the processor. LOD is important with skinned-meshes.Also splitting up the timings of Ai think routine can help, rather than having them all called at the same time.
#12
So we can do LODs on the player models as well, making them much lower poly from a distance? Didn't think about that, was focusing solely on the static models in the game in terms of cleanup and LODs... But perhaps the biggest gain could be from doing so with the player models. I'll ask Brian to take a crack at that.
Thanks again, should be a fun weekend :)
09/16/2011 (10:53 am)
Steve - thanks, I was leaning towards that... We have been LOD'ing all our objects in the game for the past month or so, gaining a frame here and there as we do so (doors, chairs, beds, etc, anything that repeats many times in the ship). Have not touched the player models though, and as I stated above was starting to think that may be an area worth looking at.So we can do LODs on the player models as well, making them much lower poly from a distance? Didn't think about that, was focusing solely on the static models in the game in terms of cleanup and LODs... But perhaps the biggest gain could be from doing so with the player models. I'll ask Brian to take a crack at that.
Thanks again, should be a fun weekend :)
#13
I agree with Steve... you need heavy LOD on characters. I expect you are spending a lot of time in mesh skinning.
09/16/2011 (11:43 am)
Use the profiler to see where you are spending your time. Holding Ctrl - F3 while in game will run the profiler during that time. On key release, you should find the profile log file in your game directory.I agree with Steve... you need heavy LOD on characters. I expect you are spending a lot of time in mesh skinning.
#14
Thanks for the good advice. Feeling a bit dumb now, since most of this seems so obvious in hindsight. Forest for the trees I guess :)
Cheers
P
09/16/2011 (11:54 am)
I feel like I've just discovered the Holy Grail or something... How did we not think of LOD'ing the player models? And Holding CTRL-F3 to get a detailed profile of what the CPU is doing, wow... Been working in T3D for over two years now and knew nothing about that.Thanks for the good advice. Feeling a bit dumb now, since most of this seems so obvious in hindsight. Forest for the trees I guess :)
Cheers
P
#15
No LODs on characters? Wow - fixing that should help tons!
09/16/2011 (1:38 pm)
I don't think there's a stock way to handle states, but you could always create an array (if you have a set number of entities) and keep track of them there.No LODs on characters? Wow - fixing that should help tons!
#16
@Richard: So does this mean T3D1.2 will have the Soldier LODs? :)
09/17/2011 (1:43 pm)
@Paul: The TAIK could be useful as it has a zones feature that will "turn-off/hide" AI in "hidden" zones, takes some work setting up but a very good system either way, might be worth looking into.@Richard: So does this mean T3D1.2 will have the Soldier LODs? :)
#18
09/17/2011 (1:49 pm)
*double post* cough!
#19
[Edit] - I asked and the Soldier Pack includes three levels of detail, along with the source art for Max. It's mentioned in the overview but it wasn't called out in the features section (that's fixed now). The Cheetah Pack has this feature as well.
09/18/2011 (9:33 am)
@CSMP - technically, I'm not the right guy to answer that question (I've been working on iTorque docs and helping to clean up some of the T3D docs, so I don't know) - but I can ask. Steve is right, the source art is in the pack - but I'm guessing you want to know if they're included with the stock soldier in 1.2. I also noticed that LOD meshes aren't specifically noted in the features for the Soldier Pack.[Edit] - I asked and the Soldier Pack includes three levels of detail, along with the source art for Max. It's mentioned in the overview but it wasn't called out in the features section (that's fixed now). The Cheetah Pack has this feature as well.
#20
Here are two things that will murder performance in T3D:
1) Lots of shapes using "Visible Mesh" as collision type;
2) Complex shapes using "Visible Mesh" as Decal type and a few (a dozen or so) Players, Vehicles or Items casting Basic Light shadows on them;
Problem #2 is by far the worst: the decal system works by generating clipped geometry for each decal, which is very nice for static decals (saves fill rate by making them as compact as possible) but it's absolutely *not* usable for real-time shadows that update every frame! Create simple collision shapes and use those for decals instead, or forgo BL shadows altogether.
The BL shadows are one huge newbie trap: we really need a faster shadow solution for BL.
09/20/2011 (3:29 pm)
Are you certain the AI is the performance bottleneck? Most of the time the biggest performance problem in T3D is collision-related.Here are two things that will murder performance in T3D:
1) Lots of shapes using "Visible Mesh" as collision type;
2) Complex shapes using "Visible Mesh" as Decal type and a few (a dozen or so) Players, Vehicles or Items casting Basic Light shadows on them;
Problem #2 is by far the worst: the decal system works by generating clipped geometry for each decal, which is very nice for static decals (saves fill rate by making them as compact as possible) but it's absolutely *not* usable for real-time shadows that update every frame! Create simple collision shapes and use those for decals instead, or forgo BL shadows altogether.
The BL shadows are one huge newbie trap: we really need a faster shadow solution for BL.
Torque Owner Jules
Something2Play
What are your machine specs like and the frame rates you are getting with and without AI?