It sort of depends on which direction you want to go in. If you're interested in deep RL as applied specifically to LLMs, I'd second another commenter's recommendation of Spinning Up from OpenAI. It hasn't been updated for a while and is a little outdated, but it provides a really nice introduction to some key ideas like PPO: https://spinningup.openai.com/en/latest/user/introduction.ht...

If you want to get more of a bird's eye view of modern deep RL in general, then HuggingFace's course is a good place to start. The course itself is kind of "all over the place"—not necessarily a bad thing, just maybe not what you want if you're looking to go super deep on a single topic. But if you want to get a look at some robotics stuff, deep Q-learning, multi-agent stuff, etc., it'll give you a nice sort of "tasting menu." As with most HuggingFace stuff, the format is really nice, it does a good job of introducing you to key ideas/projects in the ecosystem, and it has some cool project components: https://huggingface.co/learn/deep-rl-course/en/unit0/introdu...

This one looks pretty good: https://arxiv.org/pdf/2412.05265v2, although I've only skimmed it so far.