I'm not sure every programmer should start with trying to specialize in a domain. But definitely, at some point, you're reaching diminishing returns for getting better at programming.
Put another way: Getting better at the craft of programming allows you to write programs faster and better. Specializing in new domains allows you to do things you couldn't otherwise. This is the realization that made me decide to stop learning new programming languages, and instead spend my time learning new domains - I realized that learning Haskell, while incredibly fun and rewarding to my abilities as a programmer, will not let me make fundamentally different things than with Python. Learning e.g. Deep Learning, however, would.
Btw - even learning totally other skills - e.g. learning 3d modeling helps me a lot in gaining deeper knowledge of 3d than I would otherwise have - even if it's not 3d programming per se, it's still relevant. And learning new Maths is not as directly applicable or as valuable as learning new software domains, but can open up new avenues you wouldn't otherwise have, that learning "yet more programming languages" or similar won't. (It's still more valuable to work directly on relevant things, of course, but sometimes it's fun to broaden your scope in relevant ways.)