Also reusing key/values for different queries can compress the KV cache, it can be an 1000x or 10000x improvement in bandwidth if the model is trained for it.
You can see dramatic improvements in latency and throughput if there is a large shared prefix of the queries.