Hacker News Reader

Comment by sakras | Hacker News Reader

I’ve been really excited about this paper since I went to the presentation at SIGMOD for it. I think the key innovation here is that it’s the first open standard for a format that’s just a simple combination of lightweight encodings. As far as I’m aware, most columnar DBMS already have storage like this, so it’s not very novel from a technical perspective.

However, the data science/big data world has been bogged down by inefficient, clunky formats like Parquet for quite a while and so I applaud any steps towards bridging the gap between the two worlds.

View on HN

On the other hand, the popularity and language-independence of parquet has seen a fantastic explosion of tools/products/libraries that I think is really healthy.

Generally with “data stuff” you’re chained to whatever the dominant language and tools do (for the longest time, Python and Spark) and you’d have to accomodate them and their idiosyncrasies directly. Nowadays though, basically everything has a pq lib, there’s tools at all scales: DuckDB, ClickHouse, Databend, etc. The Arrow internal representation of parquet means we’ve escaped the dominance of single massive packages (e.g. pandas and spark dataframes) as the only way to deal with certain datasets.

I think it’s great what we’ve managed to get via Parquet, even if it’s a little bit clunky.

Yeah I agree with both of you. The data ... milieu is both very frustrating and very exciting at the moment, in my opinion. There is a ton of noise - lots of vendors with big marketing budgets, endless blog spam, etc. - but also quite a bit of real signal - new or remixed stuff compared with 10 or even 5 years ago, that is really useful.

Yeah the signal/noise ratio is super interesting. The advertising budgets screech real loud, but the interesting things that make it through are real interesting.

What I’m very much hoping is things stay in the current style of “mostly” language independent formats and tools. I’ve written more than enough Python and Scala/Spark and I’ve moved on that I don’t ever want to go back to the abysmal landscape that was “Python or bust”.

Reply on news.ycombinator.com