I can only speak for myself but I use the CLI and TUIs specifically because of the limitations those modalities impose on software that uses them as the primary method for interfacing with the end user. If you want a full featured UI and all that comes with it why not just use QT or GTK or modern HTML or etc?
I think the only "improvements" I'd like to see are thing such as metadata regarding what the different streams are for, an official way to communicate using structured data as opposed to raw bytes, that sort of thing. Spiritually in the same ballpark as the difference between HTML before and after CSS.
I've written about it before, but in brief the mountain of standards is worth ripping out wholesale and doing write as a semantic protocol. It's needed to solve several things that work really poorly in terminals because their unit of compatibility is cells rather than a higher primitive. It's hard to bake something on top that works well.
The need is most strong in Accessibility, Text wrapping, Scrollback handling (i.e. making the scrollback editable and not dead text), and various other bits that make writing highly functional TUIs. Coding agents tend to find all those edge cases.
This often gets push back that it's easy, to which I can point nay sayers at the various many months long issues on things in Codex, Gemini, Claude etc. on flickering, scrolling, key handline, IME, ... doing anything moderately complex involves modeling a terminal in a terminal app effectively.
Ghostty and Rex sort of do the opposite bit. If you think of it like a stack, they write a new layer underneath that transports the cells on a better multiplexer layer 0. I think a new semantic layer 2 is an approach that would be neat to work with.
Fundamentally this is the wrong abstraction and it is painful to develop modern terminal features on top of it.
We need a terminal compositor protocol. We need to leave the pty behind and replace it with a unix socket. What other compositing protocols exist in the modern tech stack that we could extend?
Anyone can feel free to reach out to me if they want to chat about it: https://bower.sh
I think there was some discussion about this last time I mentioned the idea on hn a few weeks / months back. Trawl my history to find it pehaps.
- When you write cells, you don't have an area of the screen that you can treat as the current area, which makes screen readers have to treat the screen as characters and recompose those back into words to read. You lose any rectangular area concept and treat lines in the terminal as lines that the screen reader reads. Semantically you throw away all the layout information by writing to the terminal.
- When you write stuff to the screen and then write enough lines to the screen to push lines into the scrollback, you lose the ability to change those lines in future. You can move the cursor anywhere in the screen viewport, but not anywhere outside of it. Scrollback is dead text once written there. This is fine for most apps that need append logic, but less so where you have reasons to treat the terminal as a space where you update it or gain access to things like expanding info. e.g. pick up claude code or codex and think about how you collapse a diff or command output to headers and then expand it, in the viewport, not a problem, in the scrollback it's dead text written once.
- There's a mismatch in the way that wrapping works when applied to the screen. If you're wrapping text at the screen boundary, and letting the terminal select across lines, it hits the edge of the terminal, and wraps to the next line. There's no concept of the edge of a block, because cells are the base level primitive. This affects selecting text, copying it, etc.
- Resizing terminals after stuff has written requires repainting the existing screen and then taking care about what was written to the scrollback history. E.g. if you have 200x50 and resize to 200x49, the top line just entered the scrollback and is now part of the history that you can't edit. Unless you clear the entire scrollback and repaint. Again similar problem and reasons. Resizing to 199x50 is worse, because you lose a character on each line, and push stuff into scrollback. And also the opposite 200x51, you just pulled a line from the scrollback now, so now you have to consider what to write back to that line later - what was dead is now alive again. These problems make writing terminal coding agents that rely on having interactive parts of the screen and history parts of the screen tough.
- Scrolling in general. If you look at how each terminal scrolls, and how many lines it maps to each mouse scroll wheel action, it differs per terminal, and it differs based on mouse / trackpad use as well. The events you get are not standardized in count, frequency, acceleration, ... This makes doing anything related to scrolling when you own the entire screen and scroll state a pain as you're trying to emulate the terminal's native feel and that differs a bunch. So there's a strong missing abstraction over scroll state things instead of just plain mouse events. Having a terminal and app level collaborate to tell where the scroll is up to is the approach that seems to be necessary here.
A lot of this comes from the fact that terminals are a mix of standards tracing back many years, stacked on top of each other, implemented poorly or not at all, ECMA, VT, DEC, iTerm, Kitty, OSC, ASC, DCS, SGR, ... and they all deal in cells being the basic primitive rather than the higher level semantic thing you need to do a TUI that is region aware. That's a impedance mismatch that is hard to bridge without a new standard that just sidesteps it completely.
If you look at the history of Codex / Gemini / Claude Code, they've each run into this problem a bunch over time and solved various bits of it / ignored other bits of it.
It's narrower than replacing the cell grid. A program asks for a view the way vim takes the alternate screen. If the terminal answers, the program hands it the view's files once, and from then on the two exchange six small messages inside the normal byte stream (APC frames, the same trick kitty graphics uses). No port and no extra connection, so it works over plain ssh to a box with no GPU. When the program exits you get your terminal back, and in a terminal that doesn't answer the program can fall back to plain text.
A view is either a Lua script the terminal draws itself, in its own font and theme, or a sandboxed web page. The second one is exactly the "terminals are becoming browsers" worry from upthread, and I don't think that worry is wrong, which is why the native kind exists. Going through tmux is a non-goal: a multiplexer that doesn't pass the stream through untouched just means the program sees no host and falls back.
The only host so far is bateri, the macOS terminal I'm building, and the spec is an early Apache-2.0 draft: https://github.com/bateri/pencere/blob/main/spec/draft/pence...
Do they? Last time I checked you still couldn't select text via the keyboard in the Claude Code TUI.
They clearly had this idea that they could eventually provide pretty much the entire block mode terminal experience, the entire screen being a form and the terminal smart enough to edit it locally, with a character mode terminal. Of course, it petered out in the early 1990s, after the mass market had discovered GUIs.
How i got here i find kinda interesting. Originally i was trying to leave the terminal. Using GPUI/etc frameworks. I kept running into the terminal though, because no matter how nice the UI is - the terminal still needs to work excellently.
Eventually i decided to try an abomination. To use a custom terminal which, not unlike Kitty images, allows generic apps to pass escape codes to control the UI components. Where the UI components have no restriction on "dumb terminals". So my text editor is a two part app, an editor and a tty to render the actual custom UI APC codes.
This paved the way for true smooth scrolling from within a terminal editor, as the editor effectively paginates but the tty scrolls within the buffered pages. It allows for different UI components to mix fonts and font sizes, eg popup docs being a different size than the other text, and .. well, anything i suppose.
I have to say it's working excellently and is slowly becoming my daily driver. I won't link it though, it's a fully vibe coded and even claude designed horror.
I'm surprised i've not seen more of this in the past to be honest.
The developer seems to have used AI heavily in the elisp code and it works very well.
> I'm surprised i've not seen more of this in the past to be honest.
This is what emacs is. It's a text buffer, but text can have properties like how to render, and custom values. then a bunch of commands to manipulate the buffer and it's contents.
I'll look into that, thanks!
Terminals work so well because there's a huge frictional cost to adding to the protocol. That's what keeps the protocol simple, which is what keeps multiple implementations feasible. Unlike the web, where unless you have GoogleBux you can't afford to implement it properly.
I also don't like the fact that each TUI application (btop, for instance) comes with its own color palette. So, a friend of mine developed a terminal (https://bateri.dev, macOS only for now, but the plan is to port it to Linux later) and yet another protocol where terminal SPAs can communicate over standard SSH (inspired by X over SSH, but we didn't like its drawbacks). The SPA itself is a set of Lua scripts that runs in a sandbox inside the terminal. The application on the host that communicates with the SPA can be implemented in any language (preferably Lua to keep it simple). With this approach, all SPAs (plugins) adhere to the user's color palette preferences. (There are more details that can share if you are interested.)
So far, to stress test it and help with devex, he has built a text editor (https://github.com/bateri/textpen) and a Docker/task monitor (https://github.com/bateri/dockerpen). I am also working on a plan document viewer that works within the terminal. We'd love some feedback if you are interested.