I am a PhD student in physics and I would definitely say that linear algebra is more important.
Understanding diagonalization (and more generally, spectral theory), scalar products, and norms would be a great start. Then you can pick any quantum information textbook.
N.B. My maturity level is normally only a half step above the average 12-year-old. I can play this game all day ;)