Hacker News Reader

Comment by greglindahl | Hacker News Reader

I've been using jsonlines for logfiles from about 5 years before it acquired a name.

With line delimited logfile objects, it's easy to grep for a string of interest and then only parse the lines that match -- much more efficient than parsing an entire logfile to pick out 0.01% of the lines.

That's not the usecase talked about in this article, but it is a usecase that's important to me.

View on HN

Jsonlines is also an excellent intermediary format for working with any huge datasets that are just too big to be loaded into the memory all at once, and where data is hierarchical so csv is not an option. Compared to XML you save a lot in the total file size, and it's just as easy to parse.

It's also easy to load them into Google's BigQuery: https://cloud.google.com/bigquery/docs/loading-data#supporte...

I was just about to name that as an example. Newline separated JSON is a so much simpler solution to the problem than streaming JSON parsers. They're easier to understand, and it's trivial to implement.

From a purity point of view, sure, streaming JSON is more "correct", but I wouldn't call it the better solution.

http://jsonlines.org/

> Each Line is a Valid JSON Value

This seems like the better option I agree.

Google's lead me to this one today, lol: http://ndjson.org/

I'd make fun of NIH but I'm so guilty of it too.

I've just googled what jsonlines was and it turns out i've been using it for a while in log files too where I wanted JSON objects but needed it to behave like random access memory (exactly like you described with grep). I hadn't even realised it was a formal thing. Thank you

> That's not the usecase talked about in this article, but it is a usecase that's important to me.

Yeah, and I agree with you 100%. But lot's of things that are great for log storage aren't appropriate for an API.

Together with jq, jsonlines make for great logging.

it should be trivial to write a tool to stream a json file with proper newlines

Not sure what you're getting at -- my tools that generate json logfiles do trivially add newlines where I want them.

jq's "compact" mode will reformat files that don't have newlines where I want them.

I mean that the tools to add the newlines to a compact json file with minimal whitespace are not going to be the bottle neck. I am fairly sure that grep uses a forward streaming access pattern and not random access so inserting a relatively simple command into the pipeline to insert newlines, like jq, probably isn't going to slow it down.

The tool itself wouldn't be difficult to write, or like you said, just use jq

Reply on news.ycombinator.com