No one wants to pay the Nvidia tax
Once an organization is big enough to have AI-platform or AI-infra teams, you stop being hung-up on details like CUDA; resources will be invested to make whatever abstractions the AI researchers and engineers rely on (e.g. Torch) will be made to work, and work well.
In other words their motivations and your and my motivations are different. I have limited cash, and my primary goal is expanding my ability to get more cash.
By contrast, the Google's of this world have too much cash. Their primary goal is finding things to depend the cash on (billions at a time) that can return, or at least potentially return, either mountains of more cash, or a competitive advantage.
So yes, for them, hard problems, the harder the better, are desirable.
But a software abstraction only helps to reduce lock in. It does not unlock efficiencies on itself. For that new and/or hardware hardware might be needed.