Physics, collision, animation, and certain types of compression also use a lot of math. I also found that just plain gameplay programming in 3D games required a pretty decent command of matrix math and analytic trig.

Sure, and there are a lot of other things you don't /need/, but that will make you more valuable to employers, or to your customers if you're a business owner.

Everybody has to choose things to learn and not learn, but my experience has been that putting math in the "not learn" category can make your life/career more difficult in some job markets. If most game devs decide they don't need to know math, that's just more opportunity for me, so I'm not too bent out of shape about it. :)

You don't need to know much mathematics, but a solid practical understanding of 3D geometry is a bread-and-butter skill in game programming, no matter your speciality.