It's for KV caching. In most conversations that will mean inference. But you can do reinforcement learning using sampled sequences, and you could use KV caching to speed that up too, so that would be an instance where training could get a slight boost.
doesn't training require inference? so i guess it would help there too?
Training doesn't require inference. It uses back-propagation, a different algorithm.
Backpropagation happens after some number of inferences. You need to infer to calculate a loss function to then backprop from.
Yeah but training requires the larger memory deployment data center infra