VSCode remotes can execute code on the connecting clients, so a pwned remote server can pwn your local machine.
So yeah, if you don't trust the remote machine then I agree - you probably shouldn't use it. But I don't really think that's the use-case they had in mind.