In general, are the per-layer KV caches all independent of each other?
If we split a model layer-wise for inference across X GPUs, does the cache split as well?
If we split a model layer-wise for inference across X GPUs, does the cache split as well?
No comments yet.