Hacker News Reader

Comment by codeflo | Hacker News Reader

I'm starting to get annoyed with these DeepMind publicity stunts. They don't release any code, and don't let anyone verify their results. Their chess AI beating Stockfish involved some at least a bit questionable setup. And here? I'm a big fan of TLO, but he's currently not even in the top 50 even with his main race. With Protoss, he's clearly making amateur-level mistakes. Why choose this setting to show off your supposedly superhuman AI?

(Edit: This comment was made during the stream and it just looks like my point will be addressed before the stream's even finished, they'll let the AI play against a true top Protoss. Yeah!)

View on HN

>I'm a big fan of TLO, but he's currently not even in the top 50 even with his main race

For what it's worth, this comment reminds me a lot of[1]:

> FYI, Fan Hui, the current European Go champion that Google DeepMind defeated, is ONLY a 2nd Dan Go player. The Highest Dan ranking in Go is 9th Dan!

[1] https://news.ycombinator.com/item?id=10983898

>but he's currently not even in the top 50 even with his main race.

I don't understand this part of the complaint, and it also seems to go counter to the idea that they should be releasing more data. Do we only want to see the state of these systems once they're beating everyone? I prefer to see the progress of these systems over time. Seeing an AI getting stomped on by a pro player doesn't really show us what an AI is currently capable if most people would also get stomped by a pro player.

Even so - to me this is kind of a big deal. Go is fully observable, there might be an intractable number of combinations, but the game is well defined and everything is out in the open.

In Starcraft you don't have full knowledge of the game (fog of war), there are infinitely more game paths , you have to scout, plan for the long term (macro), react tactically in the short-term, micromanage, etc. I'm actually shocked at it's performance.

Because it's still insanely impressive. I would have been blown away if the AI was merely at the platinum level. This is an AI playing starcraft. Starcraft!

>This is an AI playing starcraft. Starcraft!

Yeah, it's not like computers were ever able to play StarCraft before. Oh, wait...

The AI normally picks a set strategy and attempts that. On harder difficulty the AI gets to cheat (more income per mineral collected). The SC2 AI is extremely primitive and at best Bronze level. The AI never really "reacts" to your actions, it's basically always on a set path. DeepMind is orders of magnitude better than the Starcraft AI.

FWIW cheating AI in RTS games is extremely common. Often they don't get fog of war, or have higher income rates. RTS AI is very very hard, so this is very impressive.

  but he's currently not even in the top 50 even with his main race

Does that really matter? AFAICT he's in the top #100 [1]. For a game as complex as SC2, it's pretty significant that DeepMind is showing these kinds of results already. It's only going to improve from here right?

[1]: https://www.gosugamers.net/starcraft2/players/14064-tlo

With Protoss, he's not remotely near that. And that does matter if you intend to make huge a press release in the shape of "DeepMind AI beats top pro player". We'll see in an hour if that's what they're going for.

there is a LiquidMana coming on now ;)

I'm excited. :)

The fact that the community has been able to use the published paper to replicate all of AlphaZero's results in both chess, go (and a number of other similar combinatorial games) and get similar performance should lay your concerns to rest.

Can you provide some links?

http://zero.sjeng.org/ for AlphaZero playing Go, and http://lczero.org/ is at least par with AlphaZero Chess based off match reproductions.

LeelaChess is one example for Chess.

https://github.com/LeelaChessZero/lczero

This project actually proves the point of the root comment. The community spent tons of time on tuning/training this network, and it still routinely loses to Stockfish, which runs on inferior hardware. It illustrates that:

1. Deep mind kept a lot of information about their methods undisclosed.

2. Despite all the claims of generality, the algorithm requires insane amounts of fiddling to train. Just read their blog.

3. The hype around AlphaZero crushing every other algorithm in chess is overblown. It's competitive, but not clearly superior.

Still, Kudos to people running Lela project for doing what DeepMind should have done - describing how things really work and testing in real-life conditions.

Where?

> Their chess AI beating Stockfish involved some at least a bit questionable setup.

I believe their eventual Science paper addressed these concerns.

Which paper? How did they address it?

Science paper: http://science.sciencemag.org/content/362/6419/1140

NYT article summarizing some of the issues addressed: https://www.nytimes.com/2018/12/26/science/chess-artificial-...

They played a newer version of SF (newest version at the time the paper was written), let SF use opening book, stronger CPU for SF, non-fixed time control for SF.

In all of the games Alphastar was slower than TLO in the build order. I agree that TLO could have been faster, but this was definitely not the deciding factor.

I think the main factor was Alpha* just had incredibly accurate assessment of exactly what part of it's army it needed to defend/attack and was very careful.

What's AlphaStar's APM. Can it just move every single unit perfectly in a single frame?

it's not perfect but definitely superhuman from what I see. These are demonstration matches and afaik they want to tune the "mechanical" abilities of the agents to be somewhat in the realm of the strongest human players but it's not a trivial thing as not every action is equal and just picking a APM number would make the AI way weaker than a human in some areas (splitting marines, building/morphing lots of units ) and way stronger in other areas (multitasking)

Commentator just mentioned that the APM they've observed from AlphaStar is comparable to pro players.

Yeah, in game 4 vs Mana the commentators mentioned there are APM limitations to keep the APM within the realm of human possibility, but the decision making and multitasking are beyond even the best players.

"There is not a pro in the world that can control stalkers that way" said Artosis (a commentator).

It really isn't. Human APM is inflated and isn't true APM, humans spam click, repeat actions, make pointless actions a lot, etc. A human can have 300 APM but only make one true action per second.

DeepMind never missclicks, doesn't have to move the mouse or fingers, never spamclicks, doesn't make pointless actions, so its APM isn't inflated.

I do believe that with this progress AlphaStar with more training could beat humans even with a more realistic APM limit.

They gimp the AI to a human level APM because they want to win through superior strategy not through 30 000 APMs.

They said the capped the APM, so the game is fair from this point of view. They also said that while the agent can see the entire map, it mostly uses local views of sort. I assume because moving the camera costs actions.

https://youtu.be/cUTMhmVh1qs?t=3218

The DeepMind folks on stream just mentioned that Alphastar's inference speed is about 50ms from observation to action.

I thought they said 350ms.

E: https://youtu.be/cUTMhmVh1qs?t=3272

They also say 50ms: https://youtu.be/cUTMhmVh1qs?t=7016

Reply on news.ycombinator.com