I spent some time learning the high level concepts first—which I found to be a very useful initial orientation—but recently I've wanted to solidify my foundations and learn the math properly.

I've found the math primer (two sections: "Linear Algebra" and "Probability and Information Theory") in this free book to be excellent so far: http://www.deeplearningbook.org/ It's a little under 50 pages for both sections.

I've seen the basics of linear algebra covered in many different places, and I think this is the most insightful yet concise intro I've come across. I haven't started the probability section yet, so I can't comment on it.