A lot of the power of the terminal metaphor is its simplicity and universality. You can implement one as a purely mechanical device (as has been done numerous times), or as an application running on your graphical environment. With a GUI terminal window you can run the very same applications you'd run on a ASR-33 teletype (although the sounds and smells won't be present).
There's actually this rather common expectation that the end user is the one that is in charge of personalising their system to their exact needs.
People need to run apps via uart or any other kind of remote link that might come in the future.
There was a graphical terminal called the Blit (sold as the AT&T 5620, succeeded by the DMD 630 and then the 730 "Multi-Tasking Graphical Terminals"). They implemented windowing, multiple streams, graphics and a lot more on top of serial connections. Essentially, multiplexing streams on top of "a UART". What you describe is more or less what they did back then, minus the windowing and multi-tasking functions, because we let the host OS deal with that.
There is even a neat 5620 emulator for macOS you can download from the app store. It requires a proprietary shell running for that tty, and I never took the time to figure out exactly how it works.
https://archives.loomcom.com/3b2/documents/DMD_Terminal/ has some good docs on that, and bitsavers.org has a ton more on the 630 and 730.
I hope one day someone finds a ton of them forgotten in a warehouse in the middle of a desert in the US, because, where I live people recycle old computers with an uncanny dedication (those barbarians!).
They decided it was a bad thing (also compatibility) and now they implemented in-band ANSI escape sequences and recommend them.
But also the difficulty of running a terminal session outside the traditional Windows command line app (think VS Code terminal pane).
https://devblogs.microsoft.com/commandline/windows-command-l...
That does not mean it is efficient. If the data is on the GPU then with in-band signaling you have to copy it off of the GPU to send it to the terminal and then the terminal has to copy it back to the GPU.