They are using RingAttention that scales self attention computation linearly by number of devices by passing KV result blocks in a ring:
https://arxiv.org/abs/2310.01889 (Submitted on 3 Oct 2023)
https://arxiv.org/abs/2310.01889 (Submitted on 3 Oct 2023)
No comments yet.