Hacker News Reader

Out of curiosity, I installed fish and set my prompt to be blank[1]. Based on this article, I would expect to see the missing line feed character as the first character on the line after every command, but I didn't. Do you know why?

[1] My `fish_prompt` function just does `echo -n ''`. I also tried `true` and an empty function body, I don't know much about fish scripting.

I assume that there is control character after carriage return that deletes everything from there to the EOL, so this do not break copy functionality or "go to EOL" shortcuts.

Oh, I just saw your reply after posting my own, though you beat me to it by quite some margin of time. That's correct, fish sends ^[[K to clear to the EOL after correctly setting the cursor position via this hack to erase any evidence of the hack and for posterity of your ptty contents.

Beware of using EL when the inverse video attribute is on.

* https://unix.stackexchange.com/a/586458/5132

Fantastic rabbit hole to dive into, thanks for linking!

The "good news" is that you can't even use tput rev under fish because it saves and restores attributes each time it regains control of the tty with very few exceptions , rev not being one of them. You can't break what you don't even support!

(Seriously though, we should add rev support.)

EDIT: I just saw that you're the same person that answered that post, which very much deserved the upvote. Your knowledge of tty behavior is impressive. It is also rare to find another person that cares about the kvt - a lot of apps that manipulate terminal attributes or use any vte codes are completely broken under even the well-written kernel virtual terminals.

If you want another rabbit hole with respect to supporting attributes, consider that the Windows Terminal people are considering adding ECMA-48 standards conformance that no other terminal emulator has, the various graphic renditions from ECMA-48:1991 that no-one else has implemented in 30 years. (I saw the issue and added overline, framed, and encircled to my terminal emulator's framebuffer front-end. It will be in version 1.41 of the toolset, but that is not out yet.)

* https://github.com/microsoft/terminal/issues/6000

* https://github.com/microsoft/terminal/issues/6205

It brought me up short to be described as caring about KVTs. (-: I've mooted doing VTs in application mode code for a couple of decades, now. Here's a white paper from 2006, which was several years in.

* http://jdebp.uk./Proposals/linux-console-daemon.html

I must update that test script for Windows Terminal.

I believe you are misunderstanding. The feature ensures that your (empty) prompt is printed at the start of a line, even if the preceding command's output doesn't end with a newline.

Start up fish and run `echo -n foo` to see `foo⏎` in your terminal.

My understanding is that the ⏎ character (and some whitespace) is printed unconditionally, but is overwritten by my prompt if the previous command ended with a newline.

If there is no prompt to overwrite the ⏎, why can I not see it?

That would actually not be OK because we can't count on the prompt to overwrite n characters, and the resulting cruft would give you an incorrect representation of the output if you captured a snapshot of the terminal state (or tried to copy and select). Fish clears the line itself after finding itself at col 0 of the correct row. See my other response for more.

Nice catch and great show of initiative there! It's actually not overwritten by the prompt but rather by the "hack"/post-execution handler itself, depending on where the glyph was emitted (which in turn depends on if the command output included a trailing \n or not) when the line is cleared after guaranteeing we're at col 0 of the next line. If the glyph wrapped to this line, then it would be cleared. If it didn't, it'll show on the line above.

I think the missing clue is that the hack is followed by `^[[K` (clear line from the cursor to the right), before control is passed to `fish_prompt` to paint the prompt or whatever else.

Here's some sample output (stripped of the hack spaces and any ansii escapes):

     mqudsi@ZBOOK /m/c/U/mqudsi> echo -n hello
     hello¶
     mqudsi@ZBOOK /m/c/U/mqudsi>

`fish_prompt` isn't called until the cursor is already at column 0 and the line is clear; the "missing line ending glyph" (here ¶ rather than ⏎ because fish correctly detected that the latter is likely not supported by the font used by my terminal emulator in yet another hack) is emitted at the end of the previous line.

You can see all this and more by piping the output of a `fish --interactive` session to `xxd -C` or `hexyl` or whatever (that'll disable tty feature detection and force ansii escapes to be emitted that you can then inspect).`

(You can also verify by simply using `sleep 1` as your fish_prompt - you'd expect to see the glyph appear at least while the `sleep` was running if fish were waiting on/expecting the prompt to overwrite the glyph.)

I did indeed skip some details in the article. In the case of Zsh, the output actually ends with `\r \r` which is probably meant to handle this exact case by having a space overwrite any marker in case the prompt doesn't.

Ah, I couldn't remember what it was that we did differently from zsh, but I knew there was something.

I would imagine - depending on the terminal emulator - the zsh behavior would cause copy-paste-cruft in the form of trailing whitespace while appearing correct, and would guess this was why we changed our implementation.

Isn't this sort of quirk what the terminfo/termcap databases are for? I guess adding new entries may no longer be feasible, but an entry there like "wraps at column" would allow every interactive command line program to benefit from fixes for new terminals.

They're about 40 years ahead of you.

am: Flag whose presence means that writing a character in the last column causes the cursor to wrap to the beginning of the next line. If `am' is not present, writing in the last column leaves the cursor at the place where the character was written. Writing in the last column of the last line should be avoided on terminals with `am', as it may or may not cause scrolling to occur (see section Scrolling). Scrolling is surely not what you would intend. If your program needs to check the `am' flag, then it also needs to check the `xn' flag which indicates that wrapping happens in a strange way. Many common terminals have the `xn' flag.

Yup. Fish checks both and acts accordingly, but they're kind of useless because `xn` could mean anything.

The real problem is I'm not sure when the last boolean capability was added to the termcap standard. While new terminals continue to add and update their termcaps, the spec itself is missing a lot of features that would help enormously.

I think the whole terminfo/termcap situation could be improved a lot. For one thing, most terminal emulators just report "xterm" or "xterm-256color" as the TERM environment variable, even though different terminals that report the same thing have different capabilities. A couple terminals, like alacritty and kitty, do ship with their own terminfo files, but that causes problems if you ssh to a box that doesn't have the terminfo file for that terminal.

It can definitely be improved. The model has some significant flaws. I started again from the fundamentals and came up with a rather different model for differentiating terminal capabilities.

* http://jdebp.uk./Softwares/nosh/guide/commands/TerminalCapab...

Note, in fairness, that not having the terminfo file is not a huge problem, as terminfo is designed so that one can transport information from one system to another, and place local database entries in one's home directory if necessary.

The Z shell takes xn to have a particular meaning, although not the one that it was supposed to have.

* https://unix.stackexchange.com/questions/167582/why-zsh-ends...

It was supposed to refer to a particular behaviour of Concept terminals. In practice, it is used in many terminfo records to denote DEC pending wrap, which the terminfo commentary even itself records as being different to the Concept glitch. Yes, terminfo has highly problematic paradigms that simply do not match how terminals and their emulators actually work.

* https://news.ycombinator.com/item?id=22691989

Regarding your second link, I cannot comment on the linked item as it has been archived by HN but as an additional reference for others and possibly a fun read for you: https://bugzilla.gnome.org/show_bug.cgi?id=754596

I like your TerminalCapabilities project.

I'll see your GNOME and raise you Microsoft.

* https://github.com/microsoft/terminal/issues/3177

I wondered if anyone remembered those things. So many people seem to be hard-coding escape sequences these days, making untold amounts of work for themselves.

I've always wondered, would it be reasonable / possible for a shell to run everythhing in a virtual tty? I would love a shell when programs couldn't effect each other's output, it was easy to dynamically put programs in the background while hiding their output, etc.

Isn’t it what one would get by running screen or tmux? Am I missing something?

I guess you mean each command not affecting the very next command that follows?

The problem is i need to know in advance if i want to put something in a seperate window, but that suggests it would be possible.

reptyr can reparent a program if you only realise after the fact.

I try and remember to have that installed on the heavy-interactive-use servers I admin for when users have an "oshi-" moment because they forgot to start something under screen.

You could sort of simulate this with dtach in zsh:

    zle-line-init() if [[ $CONTEXT = start ]] LBUFFER=$zle_prefix$LBUFFER
    zle -N zle-line-init
    mkdir $HOME/dtaches
    zle_prefix='dtach -A $(mktemp -d $HOME/dtaches/XXXXX)/sock '

I'm working on a shell that does this, for exactly the reasons you gave and more. It doesn't work well enough to announce here, but you can find an index of shells (including others that have had this idea) here: https://github.com/evmar/smash/wiki/Related-projects

Oh interesting, I have been thinking about related things for Oil:

Shell as an engine for TUI or GUI https://github.com/oilshell/oil/issues/738

That came out of a discussion with someone doing a really ambitious overhaul of terminals and display servers (linked in the issue).

I believe you could do some interesting things UI-wise with a shell that doesn't run in a terminal itself. The subprocesses would require a terminal, but the shell wouldn't. For example I've heard the suggestion to always keep the prompt at the top of the screen from a few people. That seems like it would be straightforward in an HTML UI, but it probably requires some cooperation from the shell itself.

Provide APIs to allow users to write their own line editor / interactive interface https://github.com/oilshell/oil/issues/663

This is a more Emacs-like architecture for a shell, with the interface being written in a high-level language rather than C/C++. That came out of running parts of ble.sh, which is an interactive shell written in bash !! (I believe it's the biggest bash program in the world, in both number of lines and sophistication)

If you have any ideas or suggestions, I'm interested. In the interest of time, I'm cutting out the interactive parts of the project and making it more like a library that other people can build shell UIs on top of.

I collected these links awhile ago but haven't done anything with them, there is some overlap:

https://github.com/oilshell/oil/wiki/Interactive-Shell

More shells here: https://github.com/oilshell/oil/wiki/ExternalResources

I suspect xonsh runs everything in a virtual tty. Full-screen text editors fail to work left and right, like makepkg and possibly sudoedit as well.

Author here. Thanks for the fascinating insights! I've updated the post to link to your comment.

PS: I worked with @ridiculousfish up until recently, say hi from me ^^

Thanks for the post! These things absolutely should be documented and dissected in posts like your; there’s way too much voodoo in dealing with terminals.

Woo! I have the same trick in my .bashrc, love to see it spread.

If you can, please add a note that if you want to add it to bash you _have_ to use an echo/printf from PROMPT_CMD, if you add it to PS1 bash's (or readline's?) own logic will get confused as to how long the prompt is and what column the cursor is at.

(If anyone wants to try it, personally the only caveat i noticed is that when you copy a command from the terminal window any spaces not overwritten by something else will appear in the output.)

The behaviour was initially imported from zsh's PROMPT_SP:

https://github.com/fish-shell/fish-shell/issues/397

This is the kind of rabbit hole that gives me nightmares any time I even consider the idea of writing my own shell. Fish nails the UX so well that even doing half as good a job seems like a truly massive amount of work.

As someone who has written their own shell I can still recommend it as a project. Actually writing the readline implementation was much easier than I was expecting. Not suggesting there aren't edge cases I've missed but at least that is a longer term issue you can resolve as and when people report issues (and if nobody else's uses your shell then the other use case you need to worry about it your own).

I do have some recommendations however:

- first of all use a 3rd party readline implementation. This will allow you to write your language parser, any builtins you'd need (eg `cd`) and forking routines. These are the bigger / harder problems.

- then once you have a POC you're happy with, rewriting your readline implementation becomes a natural progression. By that point you know how you want your shell to look and feel.

Fish is so cool. Thank you for building it.

I will second that. We provide VM solutions on cloud marketplace and all our VMs are using fish as the default shell. Both us and our customers love it. Thanks for building it.

Why output spaces when “tput el” works?

The goal is to get a new line only when not currently at the start of a line, so the number of spaces doesn’t depend on where the cursor is.

Oh, right; I somehow completely missed the wraparound effect of the spaces.

We wouldn’t run an external command because of the overhead, compatibility, and portability, but I’ll treat your question as “why not do what tput el would do?”

It all depends on where the input cursor is when the command terminated. You need something that works with both cases (line break and no line break) because you can’t introspect the output.

tput el clears the output to the end of line so you still have to figure out if you’re on the same line as the output or if you’re on a new line. The goal of spaces is not to clear content but to invoke the wrapping behavior of the terminal which will either cause you to be on the same line only in the case that you started off at col 0 or else somewhere in the middle of the next line, but in both cases, you’re on the row you need to be on (after taking into account whether wrapping happens at max column or max column plus one).

Edit: It’s actually a lot like rounding by adding 0.5 and truncating. And the complexities are akin to trying to do it with non-standardized or different standards of doubles and dealing with epsilon and NaN :D

Yeah, I get it now; for some reason I completely misunderstood the intended effect of the spaces.

No worries! (Now try figuring out what it does and why it does it from the code that you found spread out over a few hundred lines of C interspersed with curses calls and no comments.)

If I understand it correctly, if a terminal had the “bw” attribute, you could simply do

PROMPT_COMMAND='echo -n %;tput cub1; tput cub1; tput cud1; echo -ne \\r'

But the “bw” attribute does not seem to be common, so I guess the trick with spaces is usually necessary.

I would think so?

Not directly in fish, because we manually set the tty flags and restore them after each execution (it's why your fish prompt doesn't break when you ^C in the middle of a redirect or when you accidentally output binary to the terminal), but e.g. in bash or if fish were to emit those escape codes directly. But yeah, it definitely seems like a good hack and it would cut down on the bandwidth requirements (you can notice the lag when using fish/zsh over ssh on a lossy connection).

Note that depending on the xn attribute, \r may or may not end up taking you to col 0 of the same row, depending on if the previous output caused a soft wrap but didn't include the trailing new line - you could wind up at col 0 of the previous row instead, if I'm reading it correctly!

Errr, you can disregard that bit about it not working in fish. This has nothing to do with flags/modes and would be very basic ansi escapes.

Why "tput el", when ANSI/VT100 has won?

Won? Was there a war?

So to speak, yes. There was once a competitive marketplace with all sorts of terminals featuring totally different emulations.

No use of "VT100" to describe what we have today is actually correct.

Right, like no use of "Javascript" to describe what we have today is actually correct, that being "ECMAScript", damn it.

Why can't I get everyone to stop saying "Javascript"?

No, not like that. At all, in fact.

This is an XTerm FAQ, and you are not in the analogous situation that you purport to be in. You are abusing "VT100" to describe the current state of affairs in the world; which is most definitely not a DEC VT100, not like a DEC VT100, and most definitely different in many important respects (such as colour, for starters) to a DEC VT100. It's even different to "ANSI" (again, which should be ECMA-48) in that no terminal or terminal emulator (that I know of) actually provides standard function keys (function keys being another thing that a DEC VT100 does not have). A few of them, sort of, provide the non-standard function keys from a VT220, which isn't a VT100 either.

Far from "winning", "ANSI/VT100" was superseded (by things that were better) within a few years, long ago.

* https://invisible-island.net/xterm/xterm.faq.html#what_is_vt...

I used to have access to heaps of weird real machines as well, but sadly everything now is x32/x64.