Take a look at this related work https://arxiv.org/abs/2310.11441