Well indeed. I think the ultimate point is that Git is severely lacking here. There are clear use cases (you listed some) that drive people to submodules or LFS, but they are both shitty solutions.
We need Git (or something else) to provide a third solution that actually works. I'm really hoping that JJ realises that this is a problem that needs to be solved but I'm not holding my breath.