Getting started with linear algebra | Hacker News Reader

Getting started with linear algebra

hadrienj.github.io

226 pointsghosthamlet28 comments

But https://graphicallinearalgebra.net/ is available for free and uses easier-to-read graphical syntax. Why prefer this book instead?

I've been doing the Khan Academy linear algebra course which is good too.

The original linked text is probably better for a linear algebra learner if they want to apply it (especially if they want to apply it earlier rather than later). Your linked text looks (I've read the first 5 parts) to be a good introduction but at a higher level of abstraction (category theory and abstract algebra). In fact, I think it may present a good motivating case for understanding that level of higher abstraction (while it's ostensibly about linear algebra, the ideas transfer with some adjustments to other algebras).

It could also use some polish. It's awkward to be starting a new "episode" and find that it's two pages of off-topic discussion (presenting the case for the blog, which would serve better in an introduction than in the middle of the material).

Where is the text? Or is this an advertisement for a book?

Since this might get the right eyeballs here:

1. Does anyone have a good intuitive explanation for the meaning of the transpose operation? It seems to tie into a central symmetry between columns and rows (or equations and variables) that I'm having trouble articulating for myself

2. Why does every linear algebra book have a chapter on norms? It's always so incredibly boring and esoteric and I still haven't hit a situation where it's strictly necessary. The 2-norm seem adequate for most situations and comes out naturally from the least squares problem

PS: Strang is great. Meyer's "Matrix analysis and applied linear algebra" is even better (but more advanced)

Click on the numbered headings. They don't look like links, but take you to the text for that section.

Check out Better Explained for a good intuitive guide to linear algebra (among other things): https://betterexplained.com/articles/matrix-multiplication/

For one example the notion of a generalized inner product and its induced norm is important for the derivation of conjugate gradient algorithm. My smarter friends have told me that anything involving cost functions and covariance matrices (as might arise in a log likelihood thing for a multivariate normal random variable) will probably end up with weighted norms as well, but I have likely mangled this. So to mangle things further I think that this is very important to many fields eg control or statistics

i mean, probably the sexiest application of linear algebra is machine learning and different exotic norms are very important there. sometimes you even have to learn a norm from data (metric learning).

at the very least i feel like it’s important to understand 0, 1, 2, and infinity norm.

Computer graphics is a pretty fun application as well.

Two interpretations of transpose:

- Let <-,-> denote an inner product. Then <Au, v> = <u, A^T v> is a definition of the transpose operation. Proof: (Au)^T v = u^T A^T v = <u, A^T v>

- The transpose maps a vector v to a linear functional (a kind of function) v^T, such that v^T (u) = <v, u>. Conversely, a linear functional of a vector can be transposes to a vector. In fact, it's the latter (linear functionals to vectors) that's more interesting. This fact generalises to Hilbert spaces as the Riesz representation theorem, where the consequences can be pretty amazing.

- An arbitrary linear operator L: H -> K tranposes to a mapping between their dual spaces L^* = K^* -> H^* by L*(f) = f o L.

- The outer product produces rank-one matrices, and all rank-one matrices can be expressed as outer products.

Oh yikes. So the rabbit holes goes quite deep

So the transpose is really ties to the whole idea of inner product operators/function? (I'm not sure what the exact terminology is there)

So thinking out loud a bit about the first interpretation... The inner to me represents an operator that maps vectors to a scalar that denotes the degree of "sameness" between two vectors. And it's a whole family of operators that all share certain properties. It just so happens that to write out this operator in matrix notation (for the standard inner product) we need to transpose/rotate the first vector which is yucky and why we prefer the bracket notation

It's a cool insight but it just makes it seem like a mechanical operator that flips a column into a row to make things work for one family of operators. I'm thinking more like in the case of an orthonormal matrix/basis. We know the transpose is it's inverse. It stems from the standard inner products of the columns being equal to one which is I find kinda interesting b/c the inner products suddenly become part of linear systems and not just a separate measurement of a degree of sameness.. but the headache for me is that I can picture an orthonormal basis, but I can't picture it's inverse! :)

PS: The orthonormal basis case and the transpose by extension further makes me feel that only the standard inner product seems to really map to something directly applicable in linear systems

PPS: The 3rd bullet is a bit beyond me. And the 4th is also very interesting but I'm not sure how to expand on that to build an intuition haha. I'll need to digest it further

The coordinate-free concept behind transposes is the adjoint, which shows up in multiple places in mathematics. Basically, you have a formula:

<Ax, y> = <x, By>

where <,> is some inner product, and you say that the the linear transformation A is adjoint to B. If you take the usual inner product, then A is B^T.

A previous conversation: https://news.ycombinator.com/item?id=15948153

As for 2, norms (and general inner products) show up in more theoretical parts of math that use linear algebra. Maybe an applications-focused treatment doesn't need to focus so much on it?

> Where is the text?

Look for the word "Next" to find the button that takes you to the next page. Even though this is arguably the most important thing on the page, it is rendered in a very small font, and styled to not look like a link. Can't say I understand this design decision.

Hadrien's book is so engaging I couldn't stop reading.

It's superbly illustrated. It steadily and seamlessly progresses through each topic that you don't realize you're actually learning 'deep learning'.

I'm sure this book will get lots of praise pretty soon.

Thanks for sharing it ghosthamlet. Thanks for writing it Hadrien.

Looks like some great information but you need to change the way you style the links to the different chapters. The links look like ordinary text and I had no idea that I could click them until I accidentally hovered over one.

Thanks! It was a style error. I just fixed it!

Thank you for doing this. I'm currently working with deep learning but have never had Linear Algebra in school. Resources like this one, and the 3Blue1Brown video series on YouTube are a really nice way to get started.

3B1B is by far the best resource on LA anywhere in the Web. I might be chastice for saying it but i will go as far to say that 3B1B is better than MIT's LA.

3B1B is definitely more useful for applied purposes, but the MIT LA introduces you to the topic formally, and helps you understand research papers.

link: http://www.3blue1brown.com/essence-of-linear-algebra-page/

> A tensor is a n-dimensional array with n>2

This is just one example of why it is usually better to learn stuff from more authoritative sources. (On the other hand, I can’t help but recommend this amazing video: https://m.youtube.com/watch?v=CliW7kSxxWU.)

Read Introduction to Linear Algebra by Gilbert Strang; should be all you need. There are the MIT OCW lectures that you can use too.

Obligatory remark that "reading" a math textbook, and even moreso a basic one, is an active task; you can get the intuitions from youtube videos of the 3Blue1Brown variety, but to really learn you need to follow the text closely, put question marks on passages you're not 120% comfortable with, do the proofs that are left to the reader and the recommended exercises. If you're super comfortable with the material, you should do the more challenging exercises, usually marked with asterisks.

Basically you need a pencil, a lot of scratch paper and a capable waste basket; or a whiteboard all to yourself (but I get tired from being on my feet for hours on end).

----

E: My litmus test is the Gershgorin circle theorem. Every so often I check if I can still prove it, and if I do I try to review the surrounding matrix. I will literally vote against hiring anyone who can't at least handwave why that's true.

Purely out of curiosity, what jobs is your employer trying to fill that an intuitive understanding of said theorem is required?

I took a lot of linear algebra and I can’t recall having covered it.

>My litmus test is the Gershgorin circle theorem. Every so often I check if I can still prove it, and if I do I try to review the surrounding matrix. I will literally vote against hiring anyone who can't at least handwave why that's true.

Interesting comment with no context—i.e. not hire someone for what? I don't know if I would ever not hire someone for not understanding a math problem (unless the job was heavily math related in the sphere of the problem). I would definitely not hire someone who communicates as ineffectively as you did in this comment.

Reminds me of: http://philip.greenspun.com/humor/mcdonalds-joke

> I will literally vote against hiring anyone who can't at least handwave why that's true.

I hope you're at least hiring for a machine learning engineer and not a general software engineer to build CRUD apps.

Certainly you would want a more abstract follow up for many cases. Even engineering or computer science students may have to learn about e.g. inner product spaces and orthogonal polynomials for their numerics class.

Reply on news.ycombinator.com