I'm going through the book again and revisiting the lectures. I'd argue you only need the basics high school math covered (algebra, sets, functions) as well as knowing what vectors are (but you can always google that). The genius of the book can only be uncovered if you forget everything you know about programming and approach it with an open mind about what computation can be.