Comment by radarsat1 | Hacker News Reader

Comment by radarsat1 | Hacker News Reader

That's pretty fun, thanks for sharing ;)

Actually with a machine learning approach it's not immediately obvious to me how to incorporate it. I suppose an RNN or reinforcement learning could be used, in principle, to learn good reactions to a given sequence from the opponent, but it would take so many failures to train it, which would have to be generated more or less manually. I don't see how it could be done easily, and certainly not in an "online" way without delivering a pretty terrible play experience.

Are there any good examples of computer-controlled players using an online (or offline) machine learning-based approach to player control, that doesn't suffer from these problems?

The image that comes to mind for me is a computer player that gets better and better at beating you, but the problem is that with reinforcement learning there is no guarantee that this happens reliably and in an amount of time and gradient that would provide a good playing experience -- and then you'd be faced with the problem of it getting "too good", equally not a particularly good play experience.

Perhaps something in between? A machine learning approach that helps select and maybe parameterize these "scripts"?

There have been some attempts to apply ML to NPC control: sometimes to learn how to beat the player, sometimes how to mimic them (Forza), sometimes to learn what player wants done and assist them (B&W). But none of them have crossed the chasm from novelty to killer app.

The main difficulty with combat AI is that players enjoy mastering a game and beating it. And if the game keeps getting better and beating them instead, they will end up unhappy.

Yeah I think that's a very interesting aspect. If you want, I think this speaks to a more general point about AI that I have thought about sometimes. I suspect that we enjoy interacting with machines and using machines when we understand them, and that this understanding of how a machine operates, its capacity and its behaviour, is similar to the "theory of mind" idea that we employ a model of another person when interacting with them. That is why it is so frustrating to work with a machine that changes as you use it -- it's like trying to shoot a moving target.

I really suspect that ML will find its place in helping design machines that are then static, but ML that adapts as we use it will be frustrating to work with. At least until we develop, as a society, some kind of "theory of mind" of AI, where we learn how to expect a machine to adapt as we use it. But we are not there yet.

For now, machines are still better when they act like machines. That is why I think, for example, society adapting to the presence of things like ubiquity of self-driving cars is going to be a long and scary road.

As for video games, it seems like a computer-controlled character should be somewhat predictable, even if complicated. When the ML algorithms adapt, it should present a new character to fight, not adapt an existing one that the player has already "theorized" (i.e. figured out its pattern.)

Colin McRae Rally 2.0 was released back in 2000 and used offline learning to train a driving model. Here's an interview with the engineer that made it: http://www.ai-junkie.com/misc/hannan/hannan.html

In this modern day and age grabbing telemetry from multiplayer matches might be a good way of getting training data. Particularly for games that already measure player skill. For example creating AI for Hearthstone that played like each of the skill tiers. Periodic retraining on up to date data could then keep up with things like changes in the metagame.

This is how Drivatar works in Forza Motorsport. It was first trained and tuned offline internally at Turn 10 to give opponents personality traits and driving style (Forza 2, ah, that damn M. Rossi!), then trained from the player as well and used offline (IIRC Forza 3 had AI progress vanish after the race while 4 persisted it across races. Anyway I loved how the opponents would ostensibly start to use my own tricks and lines against me, making each lap harder than the previous one), and finally starting with Forza 5 and 6 you could have your own AI that would play as you online with your own driving style while you were away, and if it was winning, you'd get some cash.

They're fighting games. Why do we need to incorporate machine learning at all? If the computer wanted to they could always play perfectly because it could react to any input perfectly.

'learning to fight against you, sort of like a human would' is a lot more fun than 'literally cheating and perfectly countering you'

Since humans are not infallible, let's leave it at 'learning to fight against you, sort of like a superhuman would'.

It would make for a formidable opponent since you can still defeat it (with a good dose of luck).

If you forced the AI to work with a substantial input lag, it could be quite complex yet still possible to defeat.

AI's are useful as training dummies. I believe Killer Instinct has quite a good AI called "Shadow Mode" that learns from your behaviour.

http://www.polygon.com/2015/5/30/8691219/killer-instinct-tea...

It probably uses a hybrid or AI and scripting techniques.

As for things like human reaction limits etc, that can built into the AI as part of parameters it needs to operate within.

Well, that's the challenge, right ?

Not making an AI that has perfect response, but a "human-like" one.

This is just me bullshitting, but I imagine you'd want to continue shipping a simple interpreter and some kind of script for the computer opponents and use a hypothetical learning system to generate the scripts separate from the game you ship. The AI code in the game would be simpler, the scripts could be easily patched, users could generate their own scripts, etc.

I think you could probably generate and hand-tune a number of situational scripts and let a learning system decide how best to chain them for particular circumstances (e.g. if the opponent is Character A, the computer is Character B, the computer's relative health is much lower, and Character A has a full super combo gauge, then it learns a particular sequence of scripts that proves effective in replays).

Reply on news.ycombinator.com