Hacker News Reader

Comment by berkle4455 | Hacker News Reader

Data Lakehouse is where you store your data on object stores and spin up a bunch of instances (cpu/memory) as needed to crunch the data on the timeline you desire. It’s incredible to me how “solutions” are continually invented that give cloud providers plenty of stages to charge for not only storage but movement and processing of your data too.

View on HN

No, that's data lake. Data lakehouse is data lake where your objects are wrapped in a "table format" like Iceberg that allows you to query (update, itd) them like they were stored in a traditional data warehouse.

The data is still stored on object storage. It’s just a more performant format.

The root of the problem is using object storage improperly.

Data Lakehouse to me is: > A Data Lakehouse open data management architecture that combines the flexibility, cost-efficiency, and scale of Data Lake with the data management and ACID transactions of Data Warehouse with Data Lake Table Formats (Delta Lake, Apache Iceberg & Apache Hudi) that enable Business Intelligence (BI) and Machine Learning (ML) on all data.

The initial concept was created by Databricks in the CIDR Paper (https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf) in 2021.

Format is the same - parquet. Only the way metadata is stored is different.

>The root of the problem is using object storage improperly.

I don't see anything improper for any Parquet-with-metadata format.

Why would you be charged for movement in this case, I thought intra datacenter traffic was free? Or you mean you get charged to update/query the object store?

As opposed to what setup exactly? What do you propose everyone uses instead of Lakehouses?

Disaggregated storage. You still have separation of compute and storage with the bandwidth and latency benefits of directly attached storage.

Object stores are a terribly inefficient way to access and store changes of data.

It's definitely inefficient w/r/t performance (reading from Object Storage will always be several times slower than reading from disk/SSD) but the point is usually to minimize cost.

Inefficient in what respect? It really depends on your access patterns, performance requirements, and the way data is structured and stored in your object store

- someone who thinks it’s often a bad idea

it's not that it is inefficient, it is that data processing approach has to be different with lakehouses.

you can't do OLTP type workload with lakehouses - meaning updating the same value in a row million times per second, because object storage is not supposed to be used in OLTP way. however you can easily do that with relational DB, just UPDATE the value and thats it. underlying RDBMS engine will keep memory buffers updated, will update pages automatically and keep WAL for durability.

for lakehouse the proper way is to setup streaming processing and use in-memory cache to do hot data processing, and once data cools down - write to lakehouse table (object storage) like once per batch (like once every few mins). and restart batch from the beginning in case of failure - for durability.

I guess a setup such as hdfs where storage and compute are colocated and not disaggregated. But that also offers similar transactional semantics to lakehouses.

Reply on news.ycombinator.com