Approximately speaking, first year undergraduate maths level is assumed, but no more. Most topics are elaborated on as required to teach the content.

You'll need to understand calculus, i.e. understand the principles behind derivatives and integrals. You certainly won't need to be proficient in manipulating them. A brief book, like Martin Gardner's updated edition of Calculus Made Easy, is the type of background that you need. A bit more specifically, having an intuition for vector calculus and partial differential equations is important.

For QM, you will need to understand what linear algebra is for, and how it uses abstraction to simplify certain types of operations. Here's a good introduction: http://betterexplained.com/articles/linear-algebra-guide/

Honestly, I can't think of anything else that you would necessarily need to know before starting, but to get the most out of it you WILL need to follow along with his working in pen-and-paper, and get used to rewatching, or looking up topics that you struggle with.