- Improvement is hard to define consistently. Sometimes, improving classification accuracy by 0.5% means reducing error by 20%, and makes economic applications that have 100x the value or frequency of use.
- Resources used in training can be amortized over billions of times the same model is reused (much more cheaply). So even achieving an epsilon improvement in the expected utility of each inference can justify a massive increase in training cost.
- Some other notions of "better results" or "less expensive" include amount of training data required, social fairness of results, memory required or power used during inference, and so on. And there are major advances in current research on each of these better formalized axes!
That last bit is what is so frustrating in reading an article like this. The author is sweeping aside with vague arguments a great deal of work that has been written and justified to a much much higher standard of rigor (not just the VCs we all like to snark about). Readers should beware of trusting a summary like this without engaging directly with the source material.