Hacker News Reader

Comment by benjaminwootton | Hacker News Reader

Date lake can be thought of a file system. Imagine 100 CSVs in folders, usually stored on S3.

Data Lakehouse involves adding things like the ability to query via SQL, the ability to update/insert/delete, transactions.

Where before people needed warehouses for BI and lakes for data science, they can now have only one approach.

It’s likely to be a big trend as data moves to this format and arrangement and the DBMS vendors like Snowflake have to play nicely with it.

View on HN

Data warehouse, data lake, data lakehouse. The data world has some terrible terminology.

You can't resell someone a file system if you don't rename it to a "data lake". You can't resell someone an indexed file system from the 80's that can be queried with SQL unless you rename it a "data lakehouse".

Wait til you dive into ETL vs ELT or even better, if you've been doing ETL since long before "ELT" was "a thing", but everyone did ETL actually in an ELT fashion...

It's not only the names, but something like 98% of the tools too, that suck.

Disclaimer - work at Snowflake. Two quick points to mention.

1. Snowflake has always used blob stores + file data + metadata. Architecturally it’s actually always been very Lakehouse-y

2. Parquet and Iceberg should be equivalent in performance and features. It’s more than playing nicely - it’s more choose your own adventure where all things are equal.

> Where before people needed warehouses for BI and lakes for data science, they can now have only one approach.

This is all very interesting, and thank you for taking the time to explain. Any good starting points for someone who would like to know more?

Databricks popularised the concept and explain it very well - https://youtu.be/g11y-kJHr3I?si=j8FAkFsIjScHv24f

It’s a technology independent pattern though.

See Databricks authors' paper: https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf

Reply on news.ycombinator.com