HNHacker News
TopNewBestAskShowJobs

philippemnoel

1,023 karma · joined February 19, 2019

Find me at https://linktree.com/philippemnoel
submissionscomments
philippemnoel··on Tau: Open-source PaaS – A self-hosted Vercel / Netlify / Cloudflare alternative
"What's with the vilification of kubernetes?" I ask myself that question every day
philippemnoel··on Putting DuckDB in Postgres to Query Iceberg
The use case for pg_lakehouse is the following:

You have historical data which you want to surface as a user-facing experience. Companies build their frontends against Postgres, and needing to also build frontends against Snowflake/DuckDB/etc. is a lot harder to maintain than just keeping your frontend built against Postgres and use pg_lakehouse to query it.

pg_lakehouse is not trying to replace data warehouses. Data warehouses are the best tool for internal analytics (i.e. a data analyst crunching large historical data to inform some business strategy decision). pg_lakehouse is designed for exposing historical data in user-facing analytics features (like a dashboard) with a good price-performance tradeoff.

Thanks for your support :)

philippemnoel··on Putting DuckDB in Postgres to Query Iceberg
I 100% agree. This is going to be a bit of a longer comment:

1) You are right that startups want all-in Postgres. To do fast analytics, we need to get columnar storage and vectorized processing. In `pg_analytics`, we implemented a Table Access Method (TAM) to try to accomplish this.

TAMs involve changing the storage format of Postgres. When we did this, we discovered there were A LOT of Postgres internals we needed to interface with. We needed to reimplement write-ahead logs (WALs), buffer caching, MVCC, JOINs, and more. Each of these projects is a significant undertaking. Hacking on Postgres is really hard: https://rhaas.blogspot.com/2024/05/hacking-on-postgresql-is-...

As a result, in pg_analytics we cut a lot of corners to ship quickly. The performance was great, but when it came to using it in any real production setting, not having these core features of Postgres meant users couldn't really use pg_analytics, and it would take us months/over a year to build any of this. Even more so, the Table Access Method API in Postgres still has a few limitations. The folks being OrioleDB are probably the most state-of-the-art implementation of a TAM, and they joined Supabase and are pushing for upstream patches to the TAM API to enable their workload.

2) At the same time, while you are right that startups want all-in Postgres, we discovered that larger companies typically have so much analytical data that they're pretty much forced to store it on object storage (e.g. S3) for cost and scalability reasons. We realized that by building pg_lakehouse, a Foreign Data Wrapper (FDW), we could avoid needing to create columnar storage in Postgres and just do work at the query engine layer and still be able to service those customers. Since pg_lakehouse is not touching Postgres storage, we inherit MVCC, WALs, buffer caching, etc.

We're a small startup. People are often surprised to find out that ParadeDB is only 4 people and is just turning one year old. There's so much we wish to do, but we have very limited resources and need to be strategic. We actually think pausing pg_analytics in order to focus on pg_lakehouse is accelerating our path to launching pg_analytics, because we'll be able to more quickly monetize pg_lakehouse, grow our resources, and have the capacity to take on those heavier engineering projects :)

Hope this helps/is interesting to you! We're excited about your interest in pg_analytics and you can rest assured it will happen in the Postgres community, whether by us or one of the other startups working on this.

philippemnoel··on Putting DuckDB in Postgres to Query Iceberg
!! Thank you, I've put up a PR to fix this
philippemnoel··on Putting DuckDB in Postgres to Query Iceberg
We talk about it more in our first post for pg_lakehouse: https://blog.paradedb.com/pages/introducing_lakehouse

This FDW does not support predicate pushdown, so performance is weak. pg_lakehouse is the same performance as DuckDB. In addition, we're supporting table formats like Apache Iceberg, more object stores, and have many other plans to improve analytics in Postgres through pg_lakehouse.

philippemnoel··on Putting DuckDB in Postgres to Query Iceberg
We'll be bringing DataFusion again in a different part of our tech stack soon... Guesses are open for where that might be!
philippemnoel··on Putting DuckDB in Postgres to Query Iceberg
What we've observed is that pg_analytics is very useful, but significantly more work to make production-ready than pg_lakehouse, and we can service ~80% of our initial users with pg_lakehouse.

We do hope to resume work on pg_analytics eventually, but given the bigger scale of the project, we've decided to postpone it for a bit longer. But stay tuned! We'll be announcing some partnerships which will help us get there faster.

philippemnoel··on Putting DuckDB in Postgres to Query Iceberg
It would be to duckdb_iceberg, yes. We're looking to keep things in the open-source :)
philippemnoel··on Putting DuckDB in Postgres to Query Iceberg
For now, pg_lakehouse is read-only. We are planning to add write support and this is a feature we would like to support. What is your use case for it?
philippemnoel··on Putting DuckDB in Postgres to Query Iceberg
The first version of it was built using Apache DataFusion and we kept the name. We're planning for it to become more than "just" DuckDB in Postgres
philippemnoel··on Putting DuckDB in Postgres to Query Iceberg
It is! They (and we) are working on fixing that so that the performance for Iceberg gets a lot better
philippemnoel··on Pg_lakehouse: A DuckDB Alternative in Postgres
Most companies that provide AGPL, including us (ParadeDB) also offer a commercial license for interested companies. Several successful software companies (Grafana, MinIO, Citus, etc.) have chosen to be AGPL to thread the needle between being true OSS while also managing to monetize their offering :)
philippemnoel··on Pg_lakehouse: A DuckDB Alternative in Postgres
Many people write Parquet files to S3 from Kafka, and read it from DuckDB, Databricks, etc. And now Postgres! It's already configured with access control, caching, etc. and is where your operational data lives
philippemnoel··on Pg_lakehouse: A DuckDB Alternative in Postgres
Iceberg support is on the horizon. The iceberg-rust crate is still nascent, but it's coming!
philippemnoel··on Tantivy – full-text search engine library inspired by Apache Lucene
Postgres's FTS is actually quite solid! You can get very far with just the built-in tsvector. The ranking could be improved, though, which was one of the reasons for creating pg_search in the first place: https://github.com/paradedb/paradedb/tree/dev/pg_search (disclaimer: I work on pg_search @ ParadeDB)
philippemnoel··on Using Postgres for Everything
This is very much a work-in-progress in the ecosystem, though. We (ParadeDB) also have `pg_lakehouse`, which is a version of `pg_analytics` designed to work over cloud object stores (AWS S3, GCS, etc.) for "cheaper storage".

I don't know if columnar engines will make it to core Postgres (I doubt it), but I do see the work of OrioleDB making it to core someday. We've joined an "alliance", alongside the core OrioleDB folks and some people from Timescale and Yandex, to push for it.

philippemnoel··on Pg_lakehouse: Query Any Data Lake from Postgres
It depends. If you're happy with Databricks, etc. you might be good. But we've seen many users want the simplicity of querying data from Postgres for analytics, especially in case of JOINing both analytics and transactional data
philippemnoel··on Pg_lakehouse: Query Any Data Lake from Postgres
Another great benefit of it indeed
philippemnoel··on Pg_lakehouse: Query Any Data Lake from Postgres
Next week (May 20-something)
philippemnoel··on Pg_lakehouse: Query Any Data Lake from Postgres
For now it is read-only, but soon will be write-supported too. You can feed data via Kafka
philippemnoel··on Pg_lakehouse: Query Any Data Lake from Postgres
I think you're spot on
philippemnoel··on Pg_lakehouse: Query Any Data Lake from Postgres
We're working on getting our extensions approved on as many platforms as possible
philippemnoel··on Pg_lakehouse: Query Any Data Lake from Postgres
Snowflake is not in the list of supported stores on Apache OpenDAL, so likely not. It might not expose its storage APIs. I doubt users of Snowflake would want a separate query engine anyways
philippemnoel··on Pg_lakehouse: Query Any Data Lake from Postgres
pgrx is indeed wonderful and we would not be able to do our work without it. Big kudos to Eric, Jubilee and rest of team!
philippemnoel··on Pg_lakehouse: Query Any Data Lake from Postgres
Are you using the iceberg-rust crate for Rust? It's a rather young project, have you found it sufficient for your needs (if using)?
philippemnoel··on Pg_lakehouse: Query Any Data Lake from Postgres
You can see performance comparison to Hydra on ClickBench: https://benchmark.clickhouse.com/ by selecting ParadeDB and Hydra. Tl;dr: It is much faster.

From a feature-set perspective, in addition to querying local disk, we can query remote object stores (S3, GCS, etc.), table format providers (Delta Lake, soon Iceberg too).

From a code perspective, we're written in Rust on top of open-source standards like OpenDAL and DataFusion, while Hydra is their own codebase built from a fork of Citus columnar, in C.

Hydra is a cool project. Hope this helps! :)

philippemnoel··on Pg_lakehouse: Query Any Data Lake from Postgres
I wish we were that clever, but it's really just the combination of "data lake" and "data warehouse", which isn't even coined by us :)
philippemnoel··on Pg_lakehouse: Query Any Data Lake from Postgres
Yeah exactly. In practice, we've inspired ourselves from the likes of Citus and others who have adopted the AGPL-3.0 license as a good compromise and have found success. It's rather comment nowadays for infra startups to use AGPL-3.0. Other noteworthy examples: MinIO, Quickwit, Lago, etc.
philippemnoel··on Pg_lakehouse: Query Any Data Lake from Postgres
Agreed, this is why we want to support both. Maybe even Apache Hudi down the line. But I hope the industry converges to a main standard rather than Snowflake/Databricks fighting for their own formats. They can differentiate on much more meaningful features
philippemnoel··on Pg_lakehouse: Query Any Data Lake from Postgres
All ParadeDB extensions are released under AGPL-3.0. We've found that it strikes the right balance between being open-source and enabling the community to adopt for free, while also protecting us from hyperscalers and enabling us to build a sustainable business. Perhaps the topic of a blog post someday :)
← PreviousPage 4 of 6Next →