Yes, there's KV cache 'Blending' see [1].
Future versions of LMCache are aiming to support this.
[1] CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion- https://arxiv.org/abs/2405.16444
Future versions of LMCache are aiming to support this.
[1] CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion- https://arxiv.org/abs/2405.16444
No comments yet.