Providers are exceptionally easy to switch. There's no moat for enterprise-level usage. There's no "market share" to gobble up because I can change a line in my config, run the eval suite, and switch immediately to another provider.

This is marginally less true for embedding models and things you've fine-tuned, but only marginally.

I find it pretty plausible they got an 80% speedup just by making optimized kernels for everything. Even when GPUs say they're being 100% utilized, there are so many improvements to be made, like:

- Carefully interleaving shared memory loading with computation, and the whole kernel with global memory loading.

- Warp shuffling for softmax.

- Avoiding memory access conflicts in matrix multiplication.

I'm sure the guys at ClosedAI have many more optimizations they've implemented ;). They're probably eventually going to design their own chips or use photonic chips for lower energy costs, but there's still a lot of gains to be made in the software.