Comment by YeGoblynQueenne | Hacker News Reader

Comment by YeGoblynQueenne | Hacker News Reader

Parent Full thread

YeGoblynQueenne

>> AlphaGo's core is built around a similar machine learning technology to Deepmind's Atari-playing system - the single, untweaked program that was able to learn superhuman play on dozens of different Atari games just by looking at the pixels, without specialization for each particular game.

Woa there. The Deepmind Atari-playing AI was too specialised for each particular game. It had a reward function that translated the score for it. It couldn't learn the importance of the game score on its own, just from "looking at the pixels" and it couldn't learn the significance of the score display and how it changed as a result of its actions on its own. All this had to be hand-coded. And if I've seen a hint that this means it couldn't really-really generalise to other games, like Deepmind claimed, then that's this bit that you report yourself:

>> Deepmind ... did reuse the widely known core insight of Monte Carlo Tree Search.

That is no mere detail. That is the crux of the matter, right there. Deepmind used their architecture to improve MCTS far enough that it could beat Lee Se-Dol. They didn't just add MCTS to their already general-game playing system.

Because the original Atari-playing AI was completely useless for playing Go, a game that doesn't have a score and looks nothing like an Atari game. So it wasn't very general at all, despite Google's and DeepMind's claims to the contrary.

I think you're wrong here. From the Nature paper describing AlphaGo:

"We also tested against the strongest open-source Go program, Pachi, a sophisticated Monte Carlo search program, ranked at 2 amateur dan on KGS, that executes 100,000 simulations per move. Using no search at all, the RL policy network won 85% of games against Pachi."

AlphaGo does use MCTS, but it seems that most of its improvements are actually coming from the deep reinforcement learning approach.

I don't know whether this is true. If AlphaGo could do just as well without search, then why did it use search at all?

But in any case, I'm not necessarily disputing that. I'm particularly refuting the claim that the AlphaGo architecture is identical to the one that learned to play Atari games and that Deepmind have advertised as a general game-playing agent.

My comment here is specifically in reply to the GP who repeated this claim, but I'll dig up the relevant link if you're interested.

Oh, fair enough. There are certainly differences; it's definitely not exact same architecture. They are both using Deep Reinforcement Learning, but e.g. AlphaGo benefits from getting an explicit representation of the board state and game rules rather than having to learn them.

Hassabis has said that in the next few months they want to try and get up to current AlphaGo performance without using any MCTS at all.

The idea is that even there's a policy network that is able to decide at some point what the best possible move is, the tree search is done to refine this choice and to "evaluate" it. This is why a value network is derived from policy network and is used in conjunction with MCTS to make sure that the moves AlphaGo picks are good ones.

It is necessary to make multiple alternatives in the tree comparable in an easy way (and nothing is better comparable than scalars). They could also go about training a network that compares two positions to decide which one is superior, but that would require much more computation. Or another alternative would possibly to learn the value somehow jointly with the action selection, but that would possibly also be harder both to train and evaluate.

Where did he say "identical"?

Noone really said that AlphaGo is identical to the Atari playing AI from a while ago. What Deepmind did say is that that agent was a general game-playing agent. If it was, then why was it not employed against Lee Sedol? Well- because it wasn't.

Reply on news.ycombinator.com