Hacker News Reader

It's a red flag that the 1.2bil model has to fit in gpu memory, happy to be provided wrong when the code drops

That's not something that DisTrO solves, but there's plenty of research in that area! See https://arxiv.org/abs/2301.11913 , https://arxiv.org/abs/2206.01288 , https://arxiv.org/abs/2304.11277 etc :)