I think Deepseek did this in the most focussed way.. their own optimization (key value caches, downscaled images for vision support etc) in their inference stack to push the boundaries in favor at a time of hardware scarcity.

It feels like OpenAI and Anthropic have a ton of room to optimizing that regard, but they choose to rather focus on moving the frontier forward than doing things that are already possible in a more efficient way.