I thought so too until found that there are quite a bit of literatures nowadays about "merging" weights, for example, this one: https://arxiv.org/pdf/1811.10515.pdf and also the OpenCLIP paper.

Is that still the case when all models have a common ancestor (i.e. finetuned) and haven’t yet overfit on new data?