How is it possible to do non-prefix KV cache? I was under the impression that the V for one token potentially depends on the V of all previous ones.
Future versions of LMCache are aiming to support this.
[1] CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion- https://arxiv.org/abs/2405.16444