For reinforcement learning, one of OpenAI's focus areas, the book by Sutton & Barto is still the standard reference: https://webdocs.cs.ualberta.ca/~sutton/book/the-book.html

Improved algorithms have been devised since it was written, see

http://karpathy.github.io/2016/05/31/rl/

and, in particular,

https://arxiv.org/abs/1502.05477