GPT-4 offered a marginal improvement over the capability of GPT-3. Depending on how you measure it, it would be hard to call it an exponential improvement. Let's say 30% just to put a number on things. A significant improvement.

GPT-3 cost 4.6 million dollars to train. According to a cursory Google search, GPT-4 cost 100 million dollars to train, about 50x as much. Is a 0.3x improvement worth 50x the cost?

That doesn't even touch on the costs of inference. Millions of queries a day requiring heavy duty compute infrastructure. How much capital are we going to waste on the next marginal improvement? Perhaps Microsoft will "only" need another 30% increase in their carbon emissions this year to sustain the growth.