I was pretty good at math in school, but I dont like math. Therefore I haven't focused on it.
I think your assumption is that people think like:
I like programming.
I'm bad at math.
Math doesn't help my programming anyways.
Instead I think people think like:
I like programming
I'm bad at math, so I don't like it.
I'm going to look at other fields to improve my programming.
I think the problem is, and the author hints at it, is that people tend to over glorify the importance of math at the expense of drowning out other fields, like prose.
In some fields like web programming, I would say that yes, my time is better spent learning how to write understandable code instead of how to solve complex equations. As a result, I probably won't ever program video games, but thats ok, games take a lot of math, and I wouldn't enjoy that anyways. It's not that I'm lazy and just don't want to get better at programming. I just want to get better at programming in ways that I enjoy the most.
With infinite time, I would certainly do it both.