Can you please elaborate on working remote with ssh? How exactly does it work and what do you do? Are you sshing into a server and work there with your VSCode GUI? I only have connected it to WSL2 yet (by simply calling code from the terminal).
Local VSCode attaches to a remote VSCode backend. So you see the remote file system, remote shells and remote extensions running in the local editor. This works for both remote hosts and containers and is cross platform; you can develop on a remote Linux host/container using your Windows desktop, for example. This is invaluable for developing inside containers with isolated tooling, for instance.
It isn't fragile; it just works and is so transparent that it's easy to get confused about where you're actually working.
Oh wait, glad I checked because ~800MB of that is just old versions of the server.