When I teach graphics to 3rd-year college undergraduates, I tell them "this is what all that point-and-vector stuff you learnt in high school was really meant for". They've usually forgotten it all, but it comes back to them quickly, and the matrix stuff on top of that isn't very hard, so I can get them up to speed pretty quickly.
Most CS degrees require linear algebra, which talks about vector spaces, gaussian elimination, diagonalization, rank, which is all useless for graphics! Too bad that course is usually relegated to the math department, which doesn't know how fun and useful a small subset of linear algebra is!