It seems like a phenomena of this sort would have to depend on the data set you are dealing with. It might be true for "all typical data sets in important domains" but it would still seem like basic point would hold.
It's almost certainly possible to break this by pathological choice of a data distribution or model/initialization/optimization scheme. But, I don't think this is interesting -- what I think is interesting is that this seems to hold true in real life, in "natural settings".
---
(^) Whatever "natural distributions" means... (which I think is a good research direction in itself).
But that doesn't mean there's a single structure determined by a single set of parameters. Quite possibly there are numerous structures with not-compatible parameter structures.
Moreover, common AI data sets share parameters in a fashion that isn't always obvious - most images on the web are photos taken by human photographers who tend to center their subject, effectively giving them different parameters than, say, security camera footage. IE, "normal data" may not mean what we imagine.