Hacker News Reader

> Curious. To me this is the worst thing you could ever do. Talking via streams to say `cat file.txt | grep ERROR | wc -l` is cool. But you could do SOOO much more, if programs would actually output structured data streams.

A lot of people have had this thought over the decades, but it hasn't really happened -- powershell exists for linux, but who's using it? The genius of the primitive representation (stringly typed tables) is that it has just enough structure to do interesting processing but not enough to cause significant mental overhead in trying to understand, memorize and reference the structure.

Case in point of the difficulties of adding more structure without wrecking immediacy of manipulation is json.

For anything with more than 1 level of nesting, I do stuff like

    blah | jq . | grep -C3 ERROR

probably a lot more than I do

    blah | jq $SOME_EXPRESSION

because it's just so much less mental overhead -- I don't have to think about indexing into some complex hierarchy and pulling out parts of it.

I'm not saying it's not possible to get out of this local optimum, but it appears to be a lot more subtle than many people seem to think. There may be an simple and elegant solution, but it seems it has so far escaped discovery. Almost five decades later, composing pipelines of weakly structured and typed bytes (that, by convention, often are line separated tables, possibly with tab or space seperated columns) is still the only high-level software re-use via composition success story of the whole computing field.

Very few use Powershell for Linux because it doesn't pre-installed on a Linux box. Otherwise you can bet that people would be using it in large numbers. And yes I would prefer your second "mental overhead" way as it involves less typing. Unfortunately powershell is more verbose than bash not less.

Powershell is unfortunately not the shining example of a shell that best leverages structured/typed input/output succinctly.

But on Windows, sysadmins use powershell heavily. Nearly every IT department that manages windows machines uses Powershell.

> Very few use Powershell for Linux because it doesn't pre-installed on a Linux box

I don't buy that. On a GNU/Linux box, there's few things that are easier than installing a new shell, if you prefer a different shell than bash it's two commands away. Bash does the job people expect it to do and would probably be _very_ alienated it they'd had to start messing around with .net gubbins.

>And yes I would prefer your second "mental overhead" way as it involves less typing

Maybe for the first time you would. Maybe if you were to accomplish this specific thing. Anything else? Have fun diving into the manpage of your shell _and_ the programs you want to use, and you better hope they share a somewhat common approach to the implemented (object) datatype or well, good luck trying to get them to talk with each other

>Powershell is unfortunately not the shining example of a shell that best leverages structured/typed input/output succinctly

I would just remove the last part, then agree with you: ">Powershell is unfortunately not the shining example of a shell"

> Nearly every IT department that manages windows machines uses Powershell

I mean, what other choice do they have there? cmd? Yeah right, if you want to loose your will to live go for it

VBScript is still used by Windows admins

>I don't buy that. On a GNU/Linux box, there's few things that are easier than installing a new shell, if you prefer a different shell than bash it's two commands away.

When you are SSH'ing into one of 10k containers for a few commands, you will only use what is already there. Bash is there and works and that is what one will use 100% of the time. No one is going to permit Powershell to be bundled to satisfy personal preferences.

You're both moving the goal posts (if powershell were superior I and countless other people would absolutely chsh it for our accounts, since we're already not using bash anyway) and not making much sense. Many sysadmins tend to spend a fair amount of time doing command line stuff and/or writing shell scripts. If powershell offered significant enough benefits for either, of course at least some companies would standardize on it, just like your hypothetical company presumably standardized on using containers to run their 10k services rather than installing some custom k8s cluster to satisfy the whims of one individual infra guy.

When one doesn't have control over the repositories used in build and service machines since they are locked down nor have control over what goes into docker images (only secured images allowed and good luck trying to get your custom tools in), one will use the stuff that is already present.

This is far more common than you think in enterprise corporations. I work at the hypothetical one, which doesn't use k8s. (yet to upgrade cloud infrastructure of native data center)

If power-shell was bundled by default in Linux distro LTS releases, a lot of sysadmins I know would start using it, since they are already familiar with it for windows and write all their scripts in the same.

Meh for argument. Its not like 100% of people are "SSHing into 10k containers to run few commands". Its probably less then 0.00001%.

> And yes I would prefer your second "mental overhead" way as it involves less typing.

1. It doesn't, just use zsh and piping into grep becomes a single character, like so:

    alias -g G='|grep -P'

2. Even apart from that I'm a bit sceptical that you can conjure up and type the necessary jq invocation in less time than you can type the fully spelled out grep line.

Yes, jq is probably the wrong example to use here. Never seen such a tool with un-rememberable CLI syntax.

Not only that but a fixed object format also "forces" me to parse the data in a particular way. Think of representing a table in JSON. The developer of the producer will have to pick either row-major or column-major representation and then that is how all consumers will see it. If that's the wrong representation for my task I will need to do gymnastics to fix that. (Or there needs to be a transposition utility command.)

Text gets around that problem, in a sense.

Obviously JSON is not suited for tabular data, but perhaps another format could be used. Ultimately, the user shouldn't care about JSONs or tabular objects.

Text just forces you into row-major, I don't see how that's getting around the problem.

IMHO, I need both text streams and metaformat. YMMV.

Just like GUIs should but usually are not gracefully, responsively scaled to user expertise, the developer experience should but usually are not gracefully, responsively scaled to the appropriate level of scaffolding to fit for purpose to the requirements defining the problem space at hand. I need more representations and abstractions, not less.

Metaformats drag in their own logistical long tail that in many use cases are wildly heavyweight for small problems. Demanding metaformats or APIs everywhere and The Only Option trades off against the REPL-like accessibility of lesser scaffolding. API-first comes with its own non-trivial balls of string; version control between caller and callee, argument parsing between versions, impedance mismatch to the kind of generative CLI's you envision, and against other API interfaces, etc.

The current unstructured primitives on the CLI, composable into structured primitives presenting as microservices or similar functions landing into a more DevOps-style landscape, etc. represents a pretty flexible toolbox that helps mitigate some of the risks in Big Design Up Front efforts that structure tends to emerge in my experience. I think of it as REPL-in-the-large.

As I gained experience I've come to appreciate and tolerate the ragged edge uncouthness of real world solutions, and lose a lot of my fanatical puritanism that veered into astronaut architecture.

> But you could do SOOO much more, if programs would actually output structured data streams.

Like even treating code and data the same, and minimizing the syntax required so you're left with clean, parseable code and data. Maybe in some sort of tree, that is abstract. Where have I heard this idea before . . .

> I think it stems from the design failure to build your app around a CLI. Instead, you should build your app around an API and generate the CLI for that API.

Now this I am fully in favor of, and IMHO, it leads to much better code all around: you can then test via the API, build a GUI via the API, etc, etc, etc.

As my sibling dead comment says you should try powershell.

Even the original post said so in the third paragraph:

> and now PowerShell where you can load .NET assemblies and pass around structured objects