Nushell already kind of has this philosophy - there is a separation between the actually scripting of tools (prints out structured json), and the visualization back to the user if it's a terminal. On top of this, you could say, have some awareness of, "If I know I'm the remote, I can pass a json list to the userdirectly and have them render client-side, rather than pretend to be a terminal".
It may be inefficient for some cases, but then you could actually say, execute the vision of mosh.
The terminal, in the abstract, is one of the best abstractions ever. The implementation is tied down to 1980s tech.
You could always spin up a "normal" terminal if you're ssh'd into some remote server to do sysadmin stuff. IMO you shouldn't be installing personal configs on there anyways (maybe allow a copy pasted .bashrc and .tmux.conf)
Another thing you could do - neovim already has neovide, you could make this native, right? You would maybe write an adapter for neovim on the remote (again, all of this is in vresion control + nix; this is how I use dotfiles; and at work I'm doing nothing fancy since I don't have control over host, it's just temrinal integration) - if I type "nvim" in the "shell" application, it actually opens up its msgpack GUI protocol.
Maybe at some point you're just re-inventing emacs though.