Neural Nets & Games

#1 Well I went to an Machine Learning for games tutorial at the GDC so I'll see if I can help.

1) Neural Nets are just one of many ways to create an AI system that can learn and adapt.

2) Black and White did use them to some extent but the teaching of a creature is based almost completly on Baesyian learning. The only other game (commercial) that was discussed as having used neural nets was called Battle Crusier 3000 (I think) which was a total bomb. I have been told by a friend of mine that Quake III used them as well.

I don't have any paper links but I might be able to help you with some of the concepts of neural nets as well as some of the other techinques I learned about if you like.

#2 I remember reading somewhere (Gamasutra?) that the Black and White creature learning system was based on recording the players reaction to each of the creature's actions and using this to determine whether the creature would perform the action next time the opportunity came up.
Try searching through the Gamasutra articles for GDC 2005.

#3 @Bruno

Using a neural net in a game is a loosing proposition. The whole point of neural nets is that you need to teach them through repetition and correction.

Your AI would need to learn from scratch (unless a pre-taught net was used in the game). The dangers are that NNs will become too good for the player eventually and at that point you would either need to reset the neural net or stop playing.

Pattern based AI is the best match for game playing, where the AI uses patterns and decision trees to determine what to do next.

#4 Hmmm, I think I wasn't clear enough in my first post. Let me put it again :

I'm writing an academic research named "Neural Nets applied to X", where X could be Data Mining, Stock Market, Industrial Automation, in my case it's GAMES.

I already know what neural nets are and how they are implemented.

I already know that Black&White use them.

What I'm looking for are posts like Arland's that points me to other games that used them.

AND

Game ideas on how learning thru repetition and correction would be a feature.

Thanks again

#5 @Bruno

Using a neural net in a game is a loosing proposition. The whole point of neural nets is that you need to teach them through repetition and correction.

Your AI would need to learn from scratch (unless a pre-taught net was used in the game). The dangers are that NNs will become too good for the player eventually and at that point you would either need to reset the neural net or stop playing.

Pattern based AI is the best match for game playing, where the AI uses patterns and decision trees to determine what to do next.

#6 @Bruno: Here you go I knew I had the link somewhere www.gameai.com/ Almost anything you could want to know about ai in games.

@Ben: You are correct as far as I know. Black and White did not use NN for creature learning but instead to teach the creature preferences. The feedback learning technique is know as Reniforcment learning the same way you would use in Psychology. I was wrong about the Baesyian learning being used for that, if Baesyian learning is used in the game it would be used for the creature to make decisions that it has not encountered before.

@Peter: Decision trees are not really a learning method execpt to chose the best predecided AI path and as such is not real AI. Also the "danger" of NNs that you talk about is a danger with any learning based AI not just NNs. This is always a question when using machine learning and generally can be solved by a function that determines if the player is out of there league so to speak and then stop learning untill the player becomes at a level with AI. The last point I will make is you would always pre-teach a NN before releasing a game, the same goes for any AI, and they do not start from scratch but with a basic set of actions that they work from and create new ones with. Of course none of this really applies to the thread because as Bruno said he is not interested in making a game but writing a paper.

For anyone if you want to discuss the possiblities of Machine Learning for Games I am willing to contribute that knowledge that I have as I am planing a game that will most likely use a combination of Genetic Algorithims and Decision trees.

#7 Does anyone have any links to pre-taught neural nets for game AI?
Im thinking that at some point that you felt was right you could turn off the learning and just keep things as they were. In a multiplayer game you might turn it back on as your audience got more sophisicated.

#8 The big problem with a pre-taught NN is that it would be specific to the game it was developed for. There would be alot of code manipulating involved in modifiying one to fit another game system. The "turning off" of a NN is something that is generally done anyways in fact some game companies have used it to develop game ai without ever releasing a learning version. Of course I am no where near an expert on this so feel free to correct me if I am worng anyone.

#9 @Arland: That sounds reasonable. I also would expect some AIs to ship pre-taught and then learn from there... either per session, or across them.

#10 The difference is basicly:

Basic AI -

If (situation) then (reaction)


Learning AI -

Do (action) then
Evaluate player's (reaction)
If (reaction = true), continue
If (reaction = false), change (action)


In other words, Basic AI is already set up to accomidate most/all situations, meanwhile, a Learning AI will do what it knows, then see if it acted desirably, and if not, change it's routine.

Not too complicated.

I have a much better idea though! ;)

- Ronixus

#11 Well...dont keep us in suspense :)

Whats your better idea? :)

#12 First off, give them use of senses. You can adhere all sorts of distingushable data to objects based on virtual sensitization. We ourselves learn from using our senses. You can program a sense of touch for a model thats moving across the ground, assign it a limited range of movement and allow it to calculate out a desired 'plan' of traversing that ground realisticly. Like if a slope is steep, they can feel their way up it and learn how to traverse it with ease.

As it goes up, and the physics are applied, it recognizes it's speed declining. It can then be set to decide how to 'deal' with certain situations as opposed to just following a path. In this case, if it desires a faster pace, it can make up for it and possibly (with use of arms, etc.) realisticly climb over the terrain.

The main obstical would be making a bot know what it can and can't do, as well as how to do it. Give them all of the options, let them know what actions are desired, and they will figure out the rest on their own.

Sight is another huge sense that shockingly hasn't been explored quite well. Now, maybe it's not such a good idea to get too detailed into the sense of sight, but better to rely on a more basic notification of . You can then program it to recognize anything that object should present, gameplay wise. With, again, a sense of touch, it could feel a tree trunk is hard with a rough texture, though soft enough to chop through. Sight can also tell them that the tree is large and brown, and as well has green leaves attached. An audible sense could help determin that it is made of wood by recognizing the sound the tree made when it was hit with an object - You might even go so far as to point out the distinction of the object hitting the tree. Another sense, smell, might signal a 'wood' scent or barely distinguishable. Heck, if you program it to test it's taste buds out (and give them the ability to bite objects), it could even learn the taste of a tree and decide if it's desirable for food. After evaluation, if it knows what makes up a tree, it's physical traits can be then appropriated to being "A Tree." Now, just give it a desire for what that tree can bare, and ways for it to figure out how to obtain the goods, and you can teach a bot to be a lumberjack! Minimize/maximize the options to the gameplays' extent.

Now, if you want a bot with more realistic sense for it's well being, you can always try to mess with other physical and emotional senses as well. Attack can become one of the farthest things from it's mind if it knows:

A) Damage is a negative effect.
B) Damaging another can be a negative effect.
C) There are other, more desirable options.

You can program a bot to recognize that it's been given a command, then allow it the desire to obey that command or ignore it, based on how it feels as well as the conditions and situation in which it's being asked.

With this, you would need a constantly changing factor of desire, an emotional factor limit based on what you want it to believe it wants. A bot who's experienced pain more frequently than another, which has come to favor reward, will less likely be willing to go mine from a spot that has been under constant attack. After a while, this same bot could figure out it can heal by purchasing food and become less fearful of the mine and concentrate more on what it has learned to desire. In this same instance, if you wanted it to, you could even program the bot to recognize when the best time to go mine, in between the attacks, is!

These are just some slight examples of virtual sensation and realization techniques I'm going to develop for use in AI, and all it requires is some mathematical processes that calculate a desired result in a limited yet modifiable range.

Again, I like to think simple, then make things complicated, not the other way around. But I've still got a while before I can get into some real AI coding.

- Ronixus

#13 I just posted this..... you guys should check it - www.garagegames.com/mg/snapshot/view.php?qid=1251

#14 I guess it takes 'em a while to approve a snapshot... that's the first I ever posted so,

Here's a demo vid of my Pure Neural bots,
www.decryptoid.com/downloads/aidemo.wmv

... and a small video of a bot running thru a small maze-like group of boxes. He spins a bit but still makes it thru!
www.decryptoid.com/downloads/maze.wmv

#15 Bruno,

You were asking about games that benefit from NN. Now, board games may not be in your focus of interest. But just in case, are you aware that NN and reinforcement learning is a big success story in backgammon circles? (As is probably explained at e.g. www.gameai.com.) If the scope of your work allows it, I think you should definitely include backgammon.

I believe the still ongoing NN boom in BG was started by the IBM reseacher Gerry Tesauro and his various BG automata, most notably TD-gammon. Temporal difference is a reinforcement learning method where you, in short, blame the network for not being able to predict the outcome of the game already in the previous move. NNs exploiting the TD-lambda algorithm changed the whole backgammon hobby in the 90s, affecting even the play of human Grand Masters quite a bit. It is probably fair to say that the breakthrough of AI was the biggest revolution in backgammon since the invention of the doubling cube.

Articles about TD-gammon:
http://www.research.ibm.com/massive/tdl.html
http://www.research.ibm.com/massive/pitdl.pdf
http://www.cs.ualberta.ca/~sutton/book/11/node2.html

Neural network is used as an "evaluation function" for backgammon positions. The input of the network is the raw encoding of the game board, and the output is the "score" given by the NN for the game position. The scoring system used in the evaluation depends on the application, of course, but in backgammon it is basically the estimated probability P of player X winning the game (or, for player Y's moves, the probability 1-P of player Y winning the game.). Each valid move candidate is assessed one at a time by the evaluation function, and the candidate most favorable for the player whose turn it is to move is chosen as the next move.

In temporal difference learning, after choosing the next move we then compare the probability estimate of this move to the probability estimate given for the previous move. The difference between these consecutive probabilities is treated as the "error" of the neural network, which is then backpropagated into the network. This method may intuitively seem unfair, because finding out valid move candidates for each turn in backgammon involves rolling two dice -- how on earth could the NN predict what is being rolled in the course of the game?? This paradox is only nominal. Statistically the assumption still holds, even if it is strictly speaking false for any single pair of consecutive moves. The same position, or positions quite similar to it, are played hundreds of times in the course of the learning process, as well as the various successive positions. As a result, neural network automatically learns quite a bit about the dice probabilities involved.

Of course, the network will not be able to learn any reliable probabilities if nothing related to the game rules is ever backpropagated into it. Hence, the winning condition is hard-coded. When either player has successfully brought all of her checkers home, the hard-coded result that this player's probability of winning was 1, is backpropagated into the network. This important exeption connects the TD-lambda algorithm to reality, or, rather, the virtual reality of backgammon rules. The hard-coded winning condition and TD-lambda together solve the so-called "delayed reward" problem of reinforcement learning and of strategy games. Isn't it surprising that the NN is never taught anything else about the game except which positions are winning/losing ones. Yet it "self-learns" quite sophisticated strategies, especially when separate NNs are used for each of the main themes of the game ("running", "contact play", "back game" ). It needs no hard-coded tactical advice from a human, and it learns best by playing against itself, starting from "scratch", i.e. with randomly initialized NN weights.

Out of nostalgy, I have personally written a somewhat loyal implementation of Tesauro's paper in ANSI C. Yeah, it works! And it is a magic feeling to see how good the bugger actually becomes after crunching numbers overnight. But make no mistake, it takes time for NN to converge towards a good evaluation function. It takes hundreds of thousands of practice games. In calendar time, this is still fast, because a modern desktop can play like 100,000 practice games per hour with all-AI players. In terms of "generations", or games, however, I'm somewhat sceptical about whether NNs can learn fast enough to be of use as a "self-adapting" AI in e.g. FPS games. A human player wants to see the AI opponent adapt quickly, while the NN might need at least several thousand backpropagation operations to change in an interesting way. But for AI applications where the AI is allowed a sufficient pre-learning session before shipping, and not really change during play, I'm optimistic about the future of NNs as game AI.

#16 LOL. Sorry, I need to take a course on web forums. There are, for instance, these funny dates for each post. My rambling might come a bit late.

But hey, isn't "my first post" always supposed to be an embarrassement of gigantic magnitude ? :)

#17 Hell with it, it was a good read.

#18 @Esko,

Thanks for the input. It came just in time for a presentation I'm doing tomorrow. :-)