This trope is older than I am. And the standard answer is equally old: you wouldn't use the remote server's emacs, you would connect to the remote server from inside your own local emacs.
Seriously people. Tramp mode is your friend. It can even two two or more hops.
It's even more seamless than using the other machine's Emacs. Sometimes I have to remind myself which things are on which machine.
Does anyone have any idea how to do this with vim? I guess it would be possible with ssh maybe?
Look at 'netrw" plugin, which has been bundled as one of the default plugins with vim for years now.