Comment by julik | Hacker News Reader

Comment by julik | Hacker News Reader

There is a point which one needs to be aware of with Parquet: if you are not stepping into it with tools well suited for it (that is: Go, C++, Java or Python codebase/runtime, mainstream platform...) you are going to have a bad time. For all its advantages, Parquet is _very_ complicated (IMO overcomplicated, on multiple fronts), and uses binary dependencies which you absolutely need to carry around (Thrift?..). Up to a certain data size the advantage of CSV is not that it is a better format (it is decidedly, absolutely worse/more ambiguous than Parquet) but the fact that you can write a valid CSV in a couple of lines of anything, and read in a couple of lines of anything. If you ask me, Parquet - for all its advantages - is a product of the Cambrian explosion of the Apache-big-data-projects. Very little in its design warrants (or requires) using things like Thrift, so I won't exclude that something less overcomplicated (and more welcoming, and requiring less dependencies) will appear on the scene at some point.

To be honest any file format that would do 1) strong typing/binary data, 2) string deduplication would capture most of the advantages of parquet, except being a column store format which helps in some cases and hurts in others.

And even using a columnar format but without all the complexity of Parquet would be comparatively easy (when writing using buffers)

At that point just start using an SQLite database as a data format.

SQLite is a row-oriented store, not a column-oriented store, ie, there are fundamental tradeoffs between the two.

But sqlite has no compression by default.

In addition, very few of these kind of serialization formats are useful for mutable data.

One thing that I have been struggling with is to find an in-memory, MUTABLE data format/data structure that can be used between different processes in shared memory. I'm not sure one exists.

I have been using SQLite DBs to that end with some amount of success. If I remember well a lot of software of yore used BDB to the same effect.

Apache Arrow?

As far as I understand it, it's more for cross-process in-memory fast access (e.g. numbers crunching), but it's mutable.

As far as I can tell, Apache Arrow is not very mutable. It basically has to reallocate the universe.

I don't need large data between processes. I need short-lived medium data ... large enough and short-lived enough to make serialization painful but mutable enough to make "stores" kind of annoying.

Use Redis. It's an in-memory data structure server, first created to have the same data structures usable and accessible across processes and languages.

Or plenty of other in-memory database products (Apache Ignite) or fast key/value (ScyllaDB) or just use RDBMS (Postgres or SQLite).

>but the fact that you can write a valid CSV in a couple of lines of anything, and read in a couple of lines of anything.

I seriously hope you are implying the usage of a library.

Hand rolled CSV implentations are generally wrong and ruin your data.

The beauty of CSV is that you don't -need- a library to access the data.

It's just there, right in front of you.

Hand rolled CSV implentations are generally wrong and ruin your data.

Yeah sure - asymptotically. Once you've gotten a handle on your data, you can take care of all the bulletproofing later.

None of this is meant to be cheeky, BTW. The learning curve and general cognitive overhead of Parquet is very considerable.

Reply on news.ycombinator.com