A lot of the hardcore game developers (which use way more math and advanced CS concepts that most other devs) didn't go to college. This stuff is totally learnable on your own.
I recommend for example "Essential mathematics for games" by Bishop and Van Verth.