As an over 40yr old, yes it's probably too late to switch. After 30 it was a shock to me getting turned down for jobs when I aced the technical interviews.
But that brings me back to my original point, you have to find a place where your personality and interests are a match, not just RandomJob.com.
If you have a higher degree in math, there is always another job out there waiting for you.