Hacker News Reader

Comment by xwowsersx | Hacker News Reader

You're totally right. I left out the important context. I'm learning linear algebra mainly for applied use in ML/AI. I don't want to skip the theory entirely, but I've found that approaching it from the perspective of how it's actually used in models (embeddings, transformations, optimization, etc.) helps me with motivation and retaining.

So I'm looking for resources that bridge the gap, not purely computational "cookbook" type resources but also not proof-heavy textbooks. Ideally something that builds intuition for the structures and operations that show up all over ML.

View on HN

Strang's Linear algebra and learning from data is extremely practical and focused on ML

https://math.mit.edu/~gs/learningfromdata/

Although if your goal is to learn ML you should probably focus on that first and foremost, then after a while you will see which concepts from linear algebra keep appearing (for example, singular value decomposition, positive definite matrices, etc) and work your way back from there

Since you're associating ML with singular value decomposition, do you know if it is possible to factor the matrices of neural networks for fast inverse jacobian products? If this is possible, then optimizing through a neural network becomes roughly as cheap as doing half a dozen forward passes.

Not sure I am following; typical neural network training via stochastic gradient descent does not require Jacobian inversion.

Less popular techniques like normalizing flows do need that but instead of SVD they directly design transformations that are easier to invert.

The idea is that you already have a trained model of the dynamics of a physical process and want to include it inside your quadratic programming based optimizer. The standard method is to linearize the problem by materializing the Jacobian. Then the Jacobian is inserted into the QP.

QPs are solved by finding the roots (aka zeroes) of the KKT conditions, basically finding points where the derivative is zero. This is done by solving a linear system of equations Ax=b. Warm starting QP solvers try to factorize the matrices in the QP formulation through LU decomposition or any other method. This works well if you have a linear model, but it doesn't if the model changes, because your factorization becomes obsolete.

Thanks. I have a copy of Strang and have been going through it intermittently. I am primarily focused on ML itself and that's been where I'm spending most of my time. I'm hoping to simultaneously improve my mathematical maturity.

I hadn't known about Learning from Data. Thank you for the link!

Reply on news.ycombinator.com