Hacker News Reader

> If you want them to only emit data in a standardized format you are implicitly cutting out the user.

This is only true if your shell just dumps the raw output as is. But it doesn't have to do that! If we have a standard structured data format for the output, then the shell can be the one to format it. The key point here is that formatting only needs to happen at the very last step of the pipeline, when it is known that the output is about to be displayed to the user.

Indeed, this is exactly what PowerShell does, and that bit does not require data + code. It just formats lists of objects into neat tables, using their metadata to generate headers.

Note also that this doesn't need to be baked into the shell itself. It can be a separate utility, that sucks in structured data, and outputs formatted user-friendly representation. So in a legacy shell, you could still do:

$ ls | fmt

and get more or less the same output that you see from ls today (but if you were to do just ls, you'd get JSON). Whereas in a new and fancy shell, you'd get |fmt appended automatically at the end.

This, by the way, is also how PowerShell does it, except that fmt is called Out-Default.

Another option is to check what stdout is hooked up to – if it is hooked up to a terminal output something human readable, if it is hooked up to a pipe output something machine readable like JSON. (Actually, a number of utilities do this terminal-vs-nonterminal check already – e.g. GNU ls with --color=auto).

One problem I've seen with this approach in practice, is sometimes you want the human readable output to go to a pipe. For example, when you use jq, it does syntax highlighting of JSON when talking to a terminal, and omits it when talking to a pipe or a file, but then you want to use a pager which understands ANSI escape sequences (e.g. less -R), so now you have to pass an option to tell the command to output ANSI escape sequences even though it is talking to a pipe not a terminal (e.g. jq -C).

I wish there was a standard mechanism in Unix for the two ends of a pipe to negotiate about what data format goes down the pipe – MIME type, character set, etc. Then if I was piping jq to less, jq could ask less "do you understand ANSI color escapes?" And less could reply "yes I do please send them". Then I wouldn't need to remember to pass -C to jq and -R to less. (Maybe this could be implemented as ioctls at each end, by which the sender transmits a list of MIME types it supports, and the receiver replies by choosing one of them...)

Less supports setting options in the LESS environment variable, so if you set LESS='-R' in your .bashrc/whatever, you don't have to pass -R to less. (If you do `jq foo | LESS='' less` for the rare times you need less to not behave that way.)

Sadly, it doesn't seem like jq will read an environment setting to always specific -C

There's an easy solution for that, although it requires a richer pipe API.

Let each pipe actor, including the terminal/shell, decide what to do. Between each stream, they can rely on a MIME type and additional metadata to make decisions. For example, if you do:

    generate_data | parse_csv | sort

Here, generate_data produces text/csv, parse_csv consumes it and emits application/json or whatever, and sort consumes it. The hypothetical "sort" command wouldn't know what to do with text/csv, so it could default to line-based text.

"sort" then emits JSON again, and rather than displaying unfriendly JSON, the terminal/shell combo would see that the final output stream is JSON and invoke a pre-defined presentation handler. For example, there could be a global handler installed that rendered JSON as a table.

Or you could insert a different formatter yourself:

    generate_data | parse_csv | sort | fancy_table_formatter

Since fancy_table_formatter emits text (probably some specific terminal MIME type so we can signal that it can include things like ANSI escape codes), the shell doesn't need to do anything except display it.

> most unix tools (ls being the exception) do double duty, they both emit output for other programs and output for the user. Which one you get and how the user output is presented being controlled by flags.

Which is where they fundamentally break the principle of "doing one thing, and doing it well". Formatting output for user should be a separate step (even if implicitly handled by your shell).

That's one place where I felt like PowerShell was way more UNIX-y than UNIX commands in that many commands are truly orthogonal.

> Which is where they fundamentally break the principle of "doing one thing, and doing it well".

unless the 'thing' here is "outputting information in the most generally usable way possible"

You just need to output formating instructions, not code.

Read that imagining "formating instructions" as data that will be consumed by a default visualizer that the shell automatically pipes at the end of your command in interactive sessions.

I'm sure there's a general architecture for such thing. If no better choice is available, it can be done with plugins.