All the stats stuff you mentioned (and more) is laid out very nicely in the second chapter of "Elements of Statistical Learning" https://web.stanford.edu/~hastie/Papers/ESLII.pdf