Hacker News Reader

Comment by atombender | Hacker News Reader

The famous "each tool does just one thing well" mantra is also a depressingly overblown myth. Among the current Unix tools, there is a ridiculous amount of overlap.

That's why we have both "ls" and "find", even though they do the same thing conceptually. "ps" has column output, but the way it formats, sorts, selects etc. columns is reinvented and not transferable to other tools such as "lsof" and "netstat", not to mention "top", which of course is just a self-updating "ps". Every tool invents filtering, sorting, formatting etc. in their own unique way. The various flags are not translatable to other tools.

I've never used Powershell, but it seems to get one thing right that Unix never bothered to. Tools should emit data, or consume data, or display data, but tools should never do all three, because they will invariably reimplement something poorly that has been done elsewhere.

View on HN

Powershell did one thing wrong, in that its tools emit objects, not data. In other words, they emit data with behavior (methods). This, in turn, imposes the CLR object and memory model on the whole thing, and makes the pipeline impossible to use between distinct processes.

The right way to do this is to pick some reasonable text-based structured interchange format - s-exprs, JSON, whatever. Actually, it wouldn't be a bad thing to have a binary protocol as well, so long as all that complexity is negotiable, and is implemented by the standard library.

The problem here is that most unix tools (ls being the exception) do double duty, they both emit output for other programs and output for the user. Which one you get and how the user output is presented being controlled by flags.

If you want them to only emit data in a standardized format you are implicitly cutting out the user.

Since direct use by a human is arguably the main use case, output formatting needs to be reindtroduced somehow. You could make some adapter that you always pipe into that handles most formatting concerns but it's unlikely that it will be truly general.

The only general way to do that is for the tool to output, along with the data, also some code that can be used to properly format the data.

So there are only three ways to solve this:

* Unix solution, with tools doing double duty and each tool having its own incompatible set of flags

* Actual programming language solution, tools emit pure data. Great for programming, bad for interactive use.

* Powershell solution, programs emit data+code: you are locked into some VM.

> If you want them to only emit data in a standardized format you are implicitly cutting out the user.

This is only true if your shell just dumps the raw output as is. But it doesn't have to do that! If we have a standard structured data format for the output, then the shell can be the one to format it. The key point here is that formatting only needs to happen at the very last step of the pipeline, when it is known that the output is about to be displayed to the user.

Indeed, this is exactly what PowerShell does, and that bit does not require data + code. It just formats lists of objects into neat tables, using their metadata to generate headers.

Note also that this doesn't need to be baked into the shell itself. It can be a separate utility, that sucks in structured data, and outputs formatted user-friendly representation. So in a legacy shell, you could still do:

$ ls | fmt

and get more or less the same output that you see from ls today (but if you were to do just ls, you'd get JSON). Whereas in a new and fancy shell, you'd get |fmt appended automatically at the end.

This, by the way, is also how PowerShell does it, except that fmt is called Out-Default.

Another option is to check what stdout is hooked up to – if it is hooked up to a terminal output something human readable, if it is hooked up to a pipe output something machine readable like JSON. (Actually, a number of utilities do this terminal-vs-nonterminal check already – e.g. GNU ls with --color=auto).

One problem I've seen with this approach in practice, is sometimes you want the human readable output to go to a pipe. For example, when you use jq, it does syntax highlighting of JSON when talking to a terminal, and omits it when talking to a pipe or a file, but then you want to use a pager which understands ANSI escape sequences (e.g. less -R), so now you have to pass an option to tell the command to output ANSI escape sequences even though it is talking to a pipe not a terminal (e.g. jq -C).

I wish there was a standard mechanism in Unix for the two ends of a pipe to negotiate about what data format goes down the pipe – MIME type, character set, etc. Then if I was piping jq to less, jq could ask less "do you understand ANSI color escapes?" And less could reply "yes I do please send them". Then I wouldn't need to remember to pass -C to jq and -R to less. (Maybe this could be implemented as ioctls at each end, by which the sender transmits a list of MIME types it supports, and the receiver replies by choosing one of them...)

Less supports setting options in the LESS environment variable, so if you set LESS='-R' in your .bashrc/whatever, you don't have to pass -R to less. (If you do `jq foo | LESS='' less` for the rare times you need less to not behave that way.)

Sadly, it doesn't seem like jq will read an environment setting to always specific -C

There's an easy solution for that, although it requires a richer pipe API.

Let each pipe actor, including the terminal/shell, decide what to do. Between each stream, they can rely on a MIME type and additional metadata to make decisions. For example, if you do:

    generate_data | parse_csv | sort

Here, generate_data produces text/csv, parse_csv consumes it and emits application/json or whatever, and sort consumes it. The hypothetical "sort" command wouldn't know what to do with text/csv, so it could default to line-based text.

"sort" then emits JSON again, and rather than displaying unfriendly JSON, the terminal/shell combo would see that the final output stream is JSON and invoke a pre-defined presentation handler. For example, there could be a global handler installed that rendered JSON as a table.

Or you could insert a different formatter yourself:

    generate_data | parse_csv | sort | fancy_table_formatter

Since fancy_table_formatter emits text (probably some specific terminal MIME type so we can signal that it can include things like ANSI escape codes), the shell doesn't need to do anything except display it.

> most unix tools (ls being the exception) do double duty, they both emit output for other programs and output for the user. Which one you get and how the user output is presented being controlled by flags.

Which is where they fundamentally break the principle of "doing one thing, and doing it well". Formatting output for user should be a separate step (even if implicitly handled by your shell).

> Which is where they fundamentally break the principle of "doing one thing, and doing it well".

unless the 'thing' here is "outputting information in the most generally usable way possible"

That's one place where I felt like PowerShell was way more UNIX-y than UNIX commands in that many commands are truly orthogonal.

You just need to output formating instructions, not code.

Read that imagining "formating instructions" as data that will be consumed by a default visualizer that the shell automatically pipes at the end of your command in interactive sessions.

I'm sure there's a general architecture for such thing. If no better choice is available, it can be done with plugins.

Agreed. One huge benefit to a text format is that you can generate small "ad hoc" snippets (e.g. echo) easily. Otherwise I'd love something like Protocol Buffers here.

You would have to be prepared for a flood of data then, if I type get-printer I get a handful of useful properties, if I add ConvertTo-Json then I get 385 lines.

"Each tool does just one thing well" doesn't mean "just one tool does one thing". In fact, ETDJOTW means that interoperable alternative tools -- loose coupling -- is more readily supported. So that, say, you can transparently migrate from unsecured rsh to secured ssh. Actually: you can even gain capabilities, since rsh is (misfeature) limited to the 512 low-numbered ports, and hence, a maximum of 512 outbound connections. SSH has no such limitation, and can scale far higher as a management tool.

(That bug, and option, was what finally convinced one old-time shop to swap in SSH for RSH for such tasks, after I identified the problem and the solution.)

A better example would be "one tools" that do many things. Rob Pike suggested Perl, though IMO that's kind of foolish: Perl is Yet Another Programming Language, and programming languages, rather by design, tend to do multiple things (not always, or not always easily: see sed as an example of both caveats -- text replacement or branching code).

Other examples are generally nexus points of complex interactions. Editors (most especially Emacs). Email management (communications protocols, format protocols, ties to DNS, anti-spam, filtering, remote-access (POP and IMAP), extensions, authentication (at various levels), encryption (at various levels). The problem domain is complex, so the tools are complex.

The tool you're very likely using right now to read this would be another case in point: the Web browser. Not just for CERN any more.

That's one interpretation, but it's not really what people mean when they refer to "one tool". The progenitor of the expression is Doug McIlroy:

    This is the Unix philosophy: Write programs that do one 
    thing and do it well. Write programs to work together. 
    Write programs to handle text streams, because that is a 
    universal interface.

Interoperability is part of DOTADIW idea, but it flows from the "one thing": If each tool does one thing, you can compose them using pipes. Of course, interoperability also implies tools can be swapped out, though as with DOTADIW, it's more of a myth than reality, because reality isn't that simple.

I didn't say that interoperability was the goal, but that it's the effect.

See Steve McConnell, Code Complete, for a treatment of modular design in software and the flexibility it provides.

(The same logic applies well outside software.)

> That's why we have both "ls" and "find"

I don't get what you're trying to say.

"ls" lists files in a specific path while "find" searches for objects in a file system that match a specific query.

Are you sure you're not confused and somehow assuming that "does just one thing well" should mean "another tool must not accomplish anything that might be remotely achievable by misusing another tool"?

Both "ls" and "find" "list files in a specific path" and "search for objects that match a specific query". There's total overlap. The difference is that "ls" is a presentation-oriented tool, whereas "find" is geared towards pipelines.

> Both "ls" and "find" "list files in a specific path"

Only if you use "find" with a query that means "get me all files matching any and all criteria as long as they are in this specific path".

That's a corner case, and a very silly one.

> There's total overlap.

A very silly and far-fetched one.

Reply on news.ycombinator.com