Do you mean training each model independently and only averaging at a late stage, in order to reduce communication overhead in the distributed scenario?
So in this case I was more thinking if there were perhaps a few directions that stood out, and instead of potentially rejecting those, consider each a microfine-tune, averaging the result at each step ala the uniform soup.
Though perhaps a stupid idea, I'm not a practitioner.