Hacker News Reader

Comment by jasonpeacock | Hacker News Reader

And here I am wrapping my results in a result object line a newb:

    {
        results: [
            { ... },
            { ... },
            { ... }
        ]
    }

View on HN

well, jsonlines works extremly well for gigabytes of short json objects. You can mix and match usual unix tools, or simple scripts with just "read" in a loop, and you can parse each json object without worrying it'll take all your memory. jq can convert arrays to json lines and back. I love the format.

Line separated JSON has its uses, for example when streaming data.

The only reason people care about newlines for a stream is because they've arbitrarily chosen to fetch bytes from the stream until the next newline sequence (readline instead of read). But you could just as easily look for a different sequence, like the ASCII record separator character which was invented exactly for this task, and then you wouldn't have to destructively strip newlines from your input.

I'll switch from \n to \x1E when cat, head, text editors, etc. start supporting the latter as a synonym for the former.

We tell our customers our streaming API uses jsonlines for documentation purposes, but we actually just decode in a loop until EOF, and 400 at the first decode error. No separators necessary at all.

Ok, but what if you want to transmit two streams? You'd get something like Interlaced-newline-delimited-JSON. And so the list of required file formats knows no end.

If you're transmitting two streams you use two streams to transmit. I don't see the problem?

Two streams over the same channel.

I don't see why this is a problem with ndjson anymore than it is a problem with other stream text encodings (like CSV).

ndjson seems to be about how to format the content of the stream, not about how to multiplex multiple streams over a single channel. Use http2, zeromq, raw tcp or whatever else you like for that.

or just:

[ { ... }, { ... }, { ... }, ]

Reply on news.ycombinator.com