9,043 karma · joined March 25, 2010
Co-Founder & CTO @ OpenAI
We dump state from the bot API each tick and send it over GRPC to a Python agent, which formats the state into a tuple of Numpy arrays. That Numpy array is passed into 5 neural networks (one per agent), each of which returns a tuple of Numpy arrays. Each tuple is decoded into a semantic action, which is then returned to the game via GRPC.
The output is a trained neural network!
Probably about time to write an update. Things have progressed quite a bit since this post :).
- The 1v1 bot played at The International used a special creep block reward (and a big if statement separating that part of the agent from the self-play trained part). It trained for two weeks.
- A 2v2 bot discovered creep blocking on its own, no special reward. It trained for four weeks.
- OpenAI Five does not have a creep blocking reward, but neither (to our knowledge) does it creep block currently. Trained for 19 days!
Cost efficiency is always important, regardless of your total resources.
The preemptibles are just used for the rollouts — i.e. to run copies of the model and the game. The training and parameter storage is not done with preemptibles.
In a hard environment, your gradients will be very noisy — but effectively no more than linear in the duration you are optimizing over, provided that you have a reasonable solution for exploration. As you scale your batch size, you can decrease your variance linearly. So you can use good ol' gradient descent if you can scale up linearly in the hardness of the problem.
This is a handwavy argument admittedly, but seems to match what we are seeing in practice.
Simulators are nice because it is possible to take lots of samples from them — but there's a limit to how many samples can be taken from the real world. In order to decrease the number of samples needed from the environment, we expect that ideas related to model-based RL — where you spend a huge number of neural network flops to learn a model of the environment — will be the way to go. As a community, we are just starting to get fast enough computers to test out ideas there.
Dota is far too complex for random search (and if that weren't true, it would say something about human capability...). See our gameplay reel for an example of some of the combos that our system learns: https://www.youtube.com/watch?v=UZHTNBMAfAA&feature=youtu.be. Our system learns to generalize behaviors in a sophisticated way.
What I personally find most interesting here is that we see qualitatively different behavior from PPO at large scale. Many of the issues people pointed to as fundamental limitations of RL are not truly fundamental, and are just entering the realm of practical with modern hardware.
We are very encouraged by the algorithmic implication of this result — in fact, it mirrors closely the story of deep learning (existing algorithms at large scale solve otherwise unsolvable problems). If you have a very hard problem for which you have a simulator, our results imply there is a real, practical path towards solving it. This still needs to be proven out in real-world domains, but it will be very interesting to see the full ramifications of this finding.
Correct, we've been playing a number of pros using the same bot played at TI. We do have a stronger version which is just two days more of training (gets a 70% win rate vs the one at TI), but haven't seen a need to test it out. We'll likely do a blog post in upcoming weeks with more stats and commentary from the pros; would be curious what people would like to know!
Incidentally, the various exploits that people used are all similar to how we actually develop the bot. We try to find areas of the strategy space it hasn't explored, and then make a small tweak to encourage it to explore that area. Lots of progress comes from removing hardcoded restrictions, which are nice to get started. So the fact there exist exploits wasn't surprising to us — what would be surprising would be exploits we couldn't fix.
1v1 has always been a proof-of-concept for us. The fun part is really 5v5, which is what we're working on now (and hiring for! ping me if you're interested: gdb@openai.com).
TensorFlow by default is built with denormals off (ftz=true), so denormals aren't relevant for the applications we're interested in. We have updated the post to indicate this — thanks for the feedback!
We'll have another blog post coming in the next few days. But as a sneak peek: we use self-play to learn everything that depends on an interaction with the opponent. Didn't need to with those that don't (e.g. fixed item builds, separately learned creep block).
Great project. We've found that the VNC Universe environments are hard for today's RL algorithms primarily due to the their async nature. We're currently working on a new set of Universe environments without VNC; I'm very happy to see others inspired by the core ideas of Universe as well.
EDIT: in fact, weights were up at launch: https://github.com/openai/generating-reviews-discovering-sen...
The main interesting thing is that none of the Amazon data was labeled, while the 232 labeled examples were.
Starting something new is incredibly hard. The default is that your company never even forms, and it's on you to overcome the activation energy. I don't think I'd have the mental fortitude to stick that out alone.
In contrast, there's nothing more motivating than working with great people (and it's hard to do better than Ilya, Sam, and Elon). Everyone brings their own core strengths to the table, and if you've picked well your own efforts will be multiplied.
(I am Greg, and am reasonably happy to be me.)
I think this reply is true. But even that aside, the startup philosophy that guides me is variance maximization. So yes, when starting something new and pouring your heart into it, you have a chance of having nothing concrete to show for yourself in a few years. But if you succeed, the magnitude will be much greater, and you'll have been a core force in making that happen.
I worked on a number of startup ideas in college, which never went anywhere. The first company I worked on that had any sort of success was Stripe (which I joined when we were 4 people). I've taken an identical approach (hard work and trying to cover the entire problem surface area) to each organization I've been a part of, and am pretty happy with recommending it as a general strategy.
Perhaps "#define CTO OpenAI" is an indication that being CTO is really operating as part of a team :).
We also think safety matters, and it should be researched in lockstep with advances in the capabilities. We have good relationships with MIRI and FHI. Our safety researchers published (together with Google Brain) a roadmap of concrete safety problems [1] and work to provide tools to prevent ML systems from being subverted [2].
No one yet knows the precise details of how AI should play out. But I'd certainly prefer that, whenever it gets close, one of the organizations actually making the advances has no incentives besides ensuring a good outcome.
[1] https://openai.com/blog/concrete-ai-safety-problems/ [2] https://github.com/openai/cleverhans
Other than SWF downloading or specific Internet-enabled environments, running offline should just work.
We'd started by adapting an existing Python driver in Twisted, implementing additional encodings and offloading to threads for calls into C libraries like zlib. We got this working reasonably on small environments like Atari, but for environments which generated many update rectangles, we started to be bitten by the GIL. I still believe that one could make Python work, but it'd take quite a lot of effort.
libvncserver is a fast C driver, but it's GPL, and doesn't have any particular support for parallelization. We wanted Universe to be usable by everyone, from hobbyists to companies, so GPL was a no-go. (We actually talked to the libvncserver maintainers, who said that they would be interested in dropping GPL restriction, but there have been far too many contributors over its long history to figure out how to do so.)
Our Go driver, based on https://github.com/mitchellh/go-vnc, has scaled quite well. It takes advantage of Go's lightweight thread model: each connection runs in its own goroutine, which makes it easy to run hundreds of connections in parallel without needing hundreds of threads.
This is actually the project I'm currently leading (goal #1 here: https://openai.com/blog/openai-technical-goals/). We should have something interesting to share in a few months.
Indeed :). Anyone who's excited, please apply! Feel free to ping me with any questions: gdb@openai.com.