Hacker News Reader

> Let’s all write parsers

And yet, at least for the Unix tools I've used, nobody did write an elaborate parser. Instead, they all ended up using newlines to represent a sequence. There were never really nested structures at all. That least-common-denominator format they were forced into ended up making input and output really easy to quickly understand.

Maybe the problem with Rest APIs is that JSON does too good of a job at making it easy to represent complex structures? Maybe we'd be better off using CSV as our data format for everything.

What trade-offs make the most sense are highly dependent on the problem being addressed. If there were ever an instance where text being the universal input and output type made sense, it would be in a Unix environment. That being said, what percentage of shell scripts in the wild would you estimate contain at least one use of grep or a regular expression in general?

Agreed. Thinking in arrays is very powerful. Simple.

Say Unix tools returned JSON. What good is that without a schema? Instead of a parser, let's all write schema transforms?

> And yet, at least for the Unix tools I've used, nobody did write an elaborate parser.

sed and awk are the elaborate parsers that are used in complex unix command line scripting.

I think what the OP is getting at is that the people writing the other command line tools didn't have to write that logic (and thankfully we can instead centralize it in a limited set of tools like sed and awk)

I may be mistaken, but I assumed that they meant that no parsing needed to be done between components, and that they just stream together, which is not really the case. You often have to use sed and awk (eleborate parsers) between different components to get the necessary portions of data from one component into the other...they don't just fit together like legos.

Yeah but sed and awk are amazing “sharp tools” for this sort of task: they have a learning curve, but 100 characters of awk can do some amazing stuff.

There's some truth to that, although there's still something to be said for a lack of recursive structure. Regular expressions can only match a flat pattern. Working with a JSON expression, even if you already have a JSON parser on hand, is often going to add more complexity than parsing a flat line of raw text using a regular expression.

> Instead, they all ended up using newlines to represent a sequence.

The biggest mistake in the design of Linux is that they allow newlines in filenames. Seriously, what the heck?!

Also, tools like jq, xsv and xmlstarlet really solve this so-called “problem”: you just need an widely-supported serialization Formats and good tools to focus on the subset of structured data and pull it out into easily-manipulated text.

> Are we all not glad we don’t use the Unix method of communicating on the web? Right? Any arbitrary command string can be the argument list for your program, and any arbitrary set of characters can come out the other end. Let’s all write parsers.

Um, but that's exactly what we do use on the web. Oh, sure, there's some really popular formats for input and output with widely available parsers (HTML, XML, JSON), and HTTP itself (as well as other protocols) specifies headers and other formats that need to be parsed before you get to those body formats, including telling you what those headers tell you about the other formats so you can no which parsers you need to write, or use if you can find one written for you.

There is also an audience mismatch. Most of the criticism of pipelines is from professional programmers.

A lot of the value of scripting, pipes and associated concepts is low barrier to entry.

I agree with the value of strong typing. But I also remember how infuriating it was to learn to work with a type system when I was learning to write code.

When I need a quick answer of some sort, iteratively piping crap at a series of "| grep | awk" is exactly what the doctor ordered. Sure, I could bullet proof it nicely and make it reusable by investing time the time to write it in something saner, but there's zero reason to - I'm not likely to ever want to perform the same action again.

> In my opinion as well, the future of software development is somewhere in the middle of this spectrum

Unfortunately this is complexity in and of itself. I don't disagree that different cases require different tools, however the split should be strongly weighted in one direction. Mostly IPC over pipes with a few exceptions, mostly REST and JSON with a few exceptions, mostly language X with a few exceptions. Everyone will have their own preferences, but I think it's important to pick a side (or at least mostly pick a side) or else you accept chaos

I think of it as a matter of scale.

- At the scale of 1,000 to 1M lines of code, authors/programmers have some control, and you're dealing with a time scale of months or years.

- At the scale of 10M to 100M lines of code, no organization is an author with complete control, and you're dealing with a time scale of decades.

In the latter case, you can't make global changes to the system, so you end up with glue in the form of pipes and textual data dumps.

You're going to have some chaos because you can't go and change every system to be consistent.

So I would say the future of software development is at BOTH ends of the spectrum, not somewhere in the middle.

-----

Good material on the subject: https://www.dreamsongs.com/Files/DesignBeyondHumanAbilitiesS...

If you look at the code in a web browser or operating system you will see this "lack of control". Once a codebase reaches a certain size, there's only so much you can do with it, and you end up with loosely coupled glue around it.

I think a reasonably minimal Debian system has to have somewhere between 100M and 1B lines of code in it, and that whole thing becomes a blob that you dump on a virtually free computer like a Raspberry Pi, etc.

I think if you look at the software that banks and airlines run on you will see the same thing. There's some code from the 60's or 70's written in some weird language at the heart of it. You don't really get to architect the system; it's more accurate to say that the system places its constraints on you.

> Are we all not glad we don’t use the Unix method of communicating on the web?

Uzbl is a collection of "web interface tools" that adhere to the Unix philosophy, that come together to create a browser.

https://www.uzbl.org/

> Are we all not glad we don’t use the Unix method of communicating on the web? Right? Any arbitrary command string can be the argument list for your program, and any arbitrary set of characters can come out the other end.

He clearly uses a different web to the one I use. >.>