Hacker News Reader

Comment by snazz | Hacker News Reader

Unix pipes are great, but having to parse plain-text streams is inefficient and error-prone. Imagine how much more powerful it would be if commands returned data in structures that could be used by other applications without parsing an ad-hoc text format.

The filesystem can be used to approximate this, like it is in Plan 9, but I think that using text as the lowest common denominator leaves a lot of potential power on the table.

View on HN

Powershell has been growing on me lately. It uses objects to pass data around and it's quite nice.

Using powershell for a month or so and then going back to doing things with text-parsing UNIX commands really highlights how much better things could be if only we moved past doing things the same way they were done in the 70s.

Have you checked out NuShell?

I'm really sad I missed the boat so hard on Powershell.

I grabbed it at the first public release (beta? alpha?), and the thing was absurdly slow. Trying to `ls` took around 10 seconds. I nope'd out and didn't touch it again.

Now I have a coworker who uses it religiously (on linux, who would have thought!) and when I'm watching over his shoulder I'm astounded by how great it is.

How do people use environment variables with powershell? I often want to do `FOO=bar ./program`, but this doesn't seem to be supported. I googled briefly, but there weren't many satisfactory answers.

    $Env:FOO='bar'; ./program

It's equivalent of `export FOO='bar'; ./program` , which is not same as `FOO='bar' ./program`, which, in turn, is equivalent to `OLD_FOO="$FOO"; export FOO='bar'; ./program; export FOO="$OLD_FOO";` .

there isn't a directly supported way. you write a function to backup, run and restore the old variable to get the same behaviour. Not very ideal.

It may be nice for small amount of data on pipeline, but is very slow compared to unix pipes. See their own documentation how they tell you to not do that: https://docs.microsoft.com/en-us/windows-server/administrati...

Interesting idea, might be important to brush up on the implementation rationale of Unix.

To that end there is The Design of The Unix Operating System, Bach [1]. Or A Fast File System for Unix [2]. Or The Design and Implementation of a Log-Structured File System [3]. It seems to me that files solve for the problem of how to deliver an economic memory management system, and they do it by paging cheap disk space so that it's an effective proxy for fast-but-expensive memory. Also files, with their file pointers, can certainly be used as data structures, but this 'understanding' of what each section means is clearly variable, so is best left to specific applications and it's not good to build into the system as a default. Different applications will want different structures for optimal performance...

[1] https://www.amazon.com/Design-UNIX-Operating-System/dp/01320...

[2] https://people.eecs.berkeley.edu/~brewer/cs262/FFS.pdf

[3] https://people.eecs.berkeley.edu/~brewer/cs262/LFS.pdf

Unix-like pipes are not restricted to sending text, they may send general bytestreams. Modern data serialization formats (flatbuffers, capnproto, msgpack, etc.) are highly efficient, and using serialization makes it easier to deploy applications in a distributed setting.

Sure, but the standard GNU or BSD userland doesn't do that, does it? If you're going to the trouble of implementing your own userspace, then you might as well use a consistent, proper data serialization format (either text or binary, but text is more accessible).

The value is in having everything speak a common standard. In unix, that 'standard' is lines of text, which is inherently linear and lacks the ability to idiomatically represent hierarchy.

Conceivably, one could use modern serialization formats or we could do something a bit more interesting, like supporting some notion of "objects" directly in the operating system or runtime (in the case of powershell, the "runtime" is the .net vm; in unix it's the OS).

Format:

* Field type: S - 16 bytes, D - 32, L - 64, Q - 128, B - 256, P - 512, T - 1024.

* Field ID - 3 bytes ASCII Letters, numbers, '_';

* Field-value separator - 2 bytes: ':' and field value terminator, e.g. ': ' or ':"' or ':[' or ':{' or ':(' ;

* Field value 9 bytes ASCII for S, 27 for D, and so on, padded with ' ';

* Field separator: 1 byte '\n';

  SOBJ: X23456789\n
  SFN_:"Length"  \n
  SFV_: 23       \n
  SEND:          \n

Such text can be parsed by both human and computer, because computer can just jump from field to field without parsing everything in between, like humans do.

You don't need to send text (but you should by default for 99%+ cases when the performance is good enough and use the advantages of a human-readable format)

The simplicity of a pipe is a feature.

Ultimately, these "text" formats are simply a matter of convention that makes some binary-formatted streams easier to document for humans' benefit. There's nothing wrong with a binary format if it's properly documented in some way, and modern serialization formats can be augmented with schemas and other features that help provide this.

Being able to easily read and edit them matters too (pipes are just files so one might choose that file to be a persistent file on disk).

Having a none-text format would mean all your file editing tools would have to then understand this new binary format. So you’d have to rewrite every tool to manipulate that data as if it’s text. By which point you’ve effectively just reinvented ASCII.

Reply on news.ycombinator.com