Comment by shadowgovt | Hacker News Reader

Comment by shadowgovt | Hacker News Reader

Parent Full thread

I think piping works well because it's an opinionated framework for IPC. The strong opinions it holds are:

1) data is a stream of bytes

2) data is only a stream of bytes

That's it. And it turns out that's a pretty powerful abstraction... Except it requires the developer to write a lot of code to massage the data entering and/or leaving the pipe if either end of it thinks "stream of bytes" means something different. In the broad-and-flat space where it's most useful---text manipulation---it works great because every tool agrees what text is (kind of... Pipe some Unicode into something that only understands only ASCII and you're going to have a lousy day). When we get outside that space?

So while, on the one hand, it allows a series of processes to go from a text file to your audio hardware (neat!), on the other hand, it allows you to accidentally pipe /dev/random directly into your audio hardware, which, here's hoping you don't have your headphones turned all the way up.

This example also kind of handwaves something, in that you touched on it directly but called it a feature, not a bug: pipes are almost always the wrong tool if you do want structure. They're too flexible. It's way the wrong API for anything where you cannot afford to have any mistakes, because unless you include something in the pipe chain to sanity-check your data, who knows what comes out the other end?

A bit of a tangent, but... audio hardware can't really be treated as a stream of bytes either.

You used to be able to cat things to /dev/dsp, but- that used something like 8khz, 8-bit, mono audio. That's horrendous. Because, with just a stream of bytes, you have to settle for the least common denominator- /dev/dsp had IOCTLs to set sample rate, number of channels, and bit depth, but... with just a stream of bytes, you can't do that.

Similarly, video data via /dev/fb0 - AFAIK you don't even have defaults there to rely on, to display anything useful you need to do IOCTLs to find out about its format.

When do you not want structure? Seriously-

Plain, human-readable ASCII text is maybe a candidate - but even then there's implicit structure (things like handling CR/LF, tabs...)

Unicode text? You know that's structure. (Ever had a read() call return in the middle of a UTF-8 multi-byte sequence?

CSV? That's structure.

Tab-separated columns? That's structure.

Fixed-width columns? Also structure.

You don't get to not have structure. Structure is always there. The question is whether you get to have a spec for your structure, or whether it's just "well, from the output it looks like column 3 here is always the hostname of the server I want, so I'll use 'cut' or 'awk' to extract it". That approach can work in practice, but...

Yes, the structure is always there. The trick with unix pipes is that interpreting the structure is always on the receiving side. I don't care if this is CSV or TSV if all I want is the number of lines. I don't care about concrete representations such as JSON or YAML if all I want is to change one string (like a hardcoded server name) to another (like a variable reference). Can you see that? How about this: why would one need a spec or schema in order to pipe an XML file through gzip?

Yeah, I think people underestimate the value of an “escape hatch” when dealing with structured data: a Java-style API can be great at controlling downstream developers and easing library maintenance. But, as a “end-developer” such APIs can be absolutely miserable to discover when you’re trying to quickly pull out a bit of data.

But pretending JSON or YAML is something you can safely "change one string" on is a recipe for disaster unless you have total control of the whole pipe chain, input to output.

How is it less safe than changing them in a (scriptable) text editor?

It's not. Neither is a safe approach. A safe approach to mutating JSON is something like a small amount of JS to read in the JSON, mutate it as javascript objects, and JSON.stringify the result.

Good point. Perhaps it is more accurate to say that with pipes, the structure is purely up to correct semantic interpretation to maintain, whereas other abstractions carry structure alongside the payload as part of the spec of the abstraction.

Of course, one can always botch configuring that structure. You never really quite escape the rule of "garbage in, garbage out."

> That's it. And it turns out that's a pretty powerful abstraction

But that's not what an abstraction is! UNIX pipes are maximally low-level, as close to un-abstract as it is possible to get, except perhaps if it used bit-streams instead of byte-streams. It literally cannot get less abstract than that.

UNIX pipes are completely untyped, like a C codebase that uses only void* to pass all data. (Okay, some way of indicating end-of-stream is also needed, but it's still a good analogy.)

> pipes are almost always the wrong tool if you do want structure

Not almost the wrong tool, entirely the wrong tool.

You can't magically shoehorn types back into an untyped system when all the existing components assume untyped streams.

PowerShell's structured and typed object streams is more UNIX than UNIX: https://news.ycombinator.com/item?id=23423650

> But that's not what an abstraction is! UNIX pipes are maximally low-level

To put a spin on it, maximally low-level within it's particular domain, perhaps, sitting on top of the transistors below them, sitting on top of quantum physics.

Thinking a bit further, an abstraction appears to be the mapping in both directions between two domains. That being said, the two domains themselves, are independent, and don't appear to be the abstraction.

The verb "abstract" would mean we are mapping from one domain to the other.

I think "homomorphism" is the mathematical term for what I'm describing.

What's my point? We can always go lower. :)

> You can't magically shoehorn types back into an untyped system when all the existing components assume untyped streams.

If this were true, then a typed system couldn't exist on top of machine code and random access memory, because the existing components (i.e. CPU + random access memory) have no knowledge of the types you are speaking of.

I think the assumption is false. Types can be shoehorned onto untyped data.

The pipe is itself a huge abstraction. If you need proof, read the implementation. Buffering, open vs closed, process suspension when the pipe is empty and resumption when there is data... Pipes hide a lot of complexity under the hood, even as simple as they are.

The pipe abstracts the transfer of data, not the data being transferred.

Reply on news.ycombinator.com