Break the Sequential Dependency of LLM Inference Using Lookahead Decoding

Oh sweet, this looks like a nice boost. Pretty simple too. Surprised it hasn’t been tried before that I’m aware of at least.

New parallel decoding algorithm that trades flops for latency reduction