"... as you make more your costs come down"
I'd say dropping the price of o3 by 80% due to "engineers optimizing inferencing" is a strong sign that they're doing exactly that.
I'd say dropping the price of o3 by 80% due to "engineers optimizing inferencing" is a strong sign that they're doing exactly that.
This is marginally less true for embedding models and things you've fine-tuned, but only marginally.
- Carefully interleaving shared memory loading with computation, and the whole kernel with global memory loading.
- Warp shuffling for softmax.
- Avoiding memory access conflicts in matrix multiplication.
I'm sure the guys at ClosedAI have many more optimizations they've implemented ;). They're probably eventually going to design their own chips or use photonic chips for lower energy costs, but there's still a lot of gains to be made in the software.
Optimizing serving isn't unlikely: all of the big AI vendors keep finding new efficiencies, it's been an ongoing trend over the past two years.
They finally implemented DeepSeek open source methods for fast inference?