Wow, I'm mindblown this isn't getting more attention. This seems like a clear win for inference. Fine tuning cost for this is reasonable (around 0.01% of the original pre-training cost). And the performance wins seem fairly consistent.
I’m not aware of any other paper that has offered to increase inference LLM performance to this degree. Has there ever been one before?
At least while also:
- Maintaining output quality. The benchmarks used were somewhat narrow but so far so good.
- Improving not just query latency but also global throughput
- Not requiring more compute
- Having a relatively practical implementation and not adding big challenges and complexity
You could argue the insight is incremental, as it builds on what’s been done with parallel/jacobi decoding. Those previous results were necessary and important, but this may be the one that finally extracts real world value from the promise of parallel decoding.
Speculative decoding can reduce latency, but at the cost of using a lot more compute. The amazing thing here is latency and global throughput improvements would be realized because of the increase in efficiency.
From what I understand speculative decoding can also come with more challenges insofar as trying to maintain overall output quality.
For more details, please check out our paper and you can also see speedup saturates as the size of training data grows.