HNHacker News
TopNewBestAskShowJobs

philippemnoel

1,026 karma · joined February 19, 2019

Find me at https://linktree.com/philippemnoel
submissionscomments
philippemnoel··on Pg_lakehouse: Query Any Data Lake from Postgres
They've been moving more and more towards closed source over the years, which is a shame but I understand why. We don't offer time-series features today, but we're not ruling out adding support for it eventually if it is desired by our users.
philippemnoel··on Pg_lakehouse: Query Any Data Lake from Postgres
We agree. We plan to bring Iceberg support as a first-class citizen as soon as we can, but unfortunately the support in Rust these days is still limited. We and the community are working on it
philippemnoel··on Using ClickHouse to scale an events engine
ParadeDB and Hydra are completely different. We're tackling the same problem of bringing analytics inside Postgres, but using different approaches.

ParadeDB integrates industry standards like Arrow, Parquet, DataFusion to offer columnar storage + vectorized processing. Hydra is building on top of Citus Columnar.

You can read about our approach here: https://blog.paradedb.com/pages/introducing_analytics

philippemnoel··on Using ClickHouse to scale an events engine
I did not know that... That's very disingenuous! We'll make sure to keep this in mind as we make our own comparisons. We haven't tuned ParadeDB at all in our published ClickBench benchmarks, so it should be more representative
philippemnoel··on Using ClickHouse to scale an events engine
That's true, but we're trying to change that at ParadeDB. Postgres is still way ahead of ClickHouse in terms of operational simplicity, ease of hiring for DBAs who are used to operating it at scale, ecosystem tooling, etc. If you can patch the speed and cost efficiency of Postgres for analytics to a level comparable to ClickHouse, then you get the best of both worlds
philippemnoel··on Using ClickHouse to scale an events engine
ParadeDB founder here. You can see how we compare to other Postgres-based analytical offerings on ClickBench here: https://blog.paradedb.com/pages/introducing_analytics
philippemnoel··on Code search is hard
RUM is good, but it lacks some of the more complex features like language tokenizers, etc. that a full search engine library like Lucene/Tantivy (and ParadeDB in Postgres) offer
philippemnoel··on Code search is hard
That's wild. Quite impressive how far Postgres can be tuned. Is this all with tsvector?
philippemnoel··on Code search is hard
Tsvector is amazing and it goes a long way, but unfortunately as you say it lacks some of more complex FTS features like typo tolerance, language tokenizers, etc.
philippemnoel··on Code search is hard
Elasticsearch is good, and it does scale, but it is much more cumbersome and expensive to scale and operate than Postgres. If you use the managed service, you'll pay for the operational pain in the form of higher pricing.

The Postgres movement is strong and extensions like ParadeDB https://github.com/paradedb/paradedb are designed specifically to solve this pain point (Disclaimer: I work for ParadeDB)

philippemnoel··on Code search is hard
ParadeDB founder here. We'd love to be supported on Render, if the Render folks are open to it...
philippemnoel··on Ask HN: Best way to mirror a Postgres database to parquet?
No timeline yet, but we know it's a high-priority feature and are working hard on it. Best way would be to join our Slack (link here: https://github.com/paradedb/paradedb/blob/dev/README.md) to follow along. It will be in the coming weeks/months, though.
philippemnoel··on Ask HN: Best way to mirror a Postgres database to parquet?
ParadeDB founder here -- It is append-only for now, but we're working on upserts!
philippemnoel··on Pg_vectorize: Vector search and RAG on Postgres
I read this paper claiming that there's no fundamental limitations to Postgres' vector support down the line. Pretty excited for the work Supabase, Neon, Tembo, ourselves at ParadeDB and many others are doing to keep pushing it forward.

Definitely more and more vector production workloads coming to Postgres

Paper: https://www.cs.purdue.edu/homes/csjgwang/pubs/ICDE24_VecDB.p...

philippemnoel··on PGlite – Postgres in WASM
This is super cool. Postgres is slowly overlapping on every part of the data stack, even SQLite isn't safe!
philippemnoel··on Transforming Postgres into a Fast OLAP Database
It can't. We are open to exploring this, although we expect most of our users, especially at larger organizations, to want a separate search&analytics node, similar to how they use Elastic today
philippemnoel··on Transforming Postgres into a Fast OLAP Database
You're right. We're working on this currently. You can track the issue here: https://github.com/paradedb/paradedb/issues/717
philippemnoel··on Transforming Postgres into a Fast OLAP Database
You're exactly right. There is a mature project called delta-rs which we use. The Iceberg equivalent, iceberg-rs, is still very young. We are planning to contribute to it and eventually add support for Iceberg
philippemnoel··on Transforming Postgres into a Fast OLAP Database
Hey! Sorry I missed this.

1) You can find the code here: https://github.com/paradedb/paradedb/blob/996f018e3258d3989f...

For deltalake tables, we send the full plan to DataFusion.

2) Re: WALs and replication -- We are currently adding support for WALs

philippemnoel··on Transforming Postgres into a Fast OLAP Database
This is exactly how we think over at ParadeDB, and querying data lakes is on our roadmap :)
philippemnoel··on Transforming Postgres into a Fast OLAP Database
I'm not fully familiar with Hydra. From my understanding, they forked Citus columnar and improved it. I think the use case for just pg_analytics vs hydra is similar, i.e. fast analytics in Postgres. However, soon they'll diverge as we interoperate it with pg_bm25, our full-text search extension. We see ParadeDB as becoming a modern alternative to Elasticsearch, while Hydra appears to be positioning itself as a Postgres warehouse

In philosophy, we believe in playing into the ecosystem. We use DataFusion to avoid needing to write a vectorized query engine, Arrow to avoid needing to build an in-memory representation, and Parquet to avoid needing to build columnar disk storage. Citus columnar/Hydra appear to be working from first principles directly within Postgres, storing data in Postgres blocks, and writing vectorized execution operator by operator

philippemnoel··on Transforming Postgres into a Fast OLAP Database
1) and 2) Backups and replication are on the roadmap. It's the next major feature we're working on.

3) We store data in Parquet files, which is a heavily compressed file format for columnar data. This ensures good compression and compatibility with Arrow for in-memory columnar processing. We also hook at the executor level and route queries on deltalake tables to DataFusion query engine, which processes the data in a vectorized fashion for much faster execution

philippemnoel··on Transforming Postgres into a Fast OLAP Database
We haven't tested. Neon has a pretty unique storage architecture, so I suspect it wouldn't work out-of-the-box, but could with some modifications on our end. We're open to this type of partnership eventually.

If you decide to try it, I believe they now have a way to load in arbitrary Postgres extensions!

philippemnoel··on Transforming Postgres into a Fast OLAP Database
TPC-H has some published benchmarks on their website, but it's for specific hardware systems. It's certainly not as user-friendly and modern as ClickBench. I'm not sure if it's possible to make something better without their consent, though.
philippemnoel··on Transforming Postgres into a Fast OLAP Database
That's right! Greenplum is still active, but falling off.

We see pg_analytics as the next-generation Citus columnar, with much better performance and integration into the wider data ecosystem via Delta Lake, and eventually Iceberg

philippemnoel··on Transforming Postgres into a Fast OLAP Database
ParadeDB maker here -- DataFusion is amazing! Apple used it to rebuild part of Spark and open-sourced it under Comet too: https://github.com/apache/arrow-datafusion-comet/pull/1

It's a lot more mature than people think :)

philippemnoel··on Transforming Postgres into a Fast OLAP Database
One of the ParadeDB makers here -- Yes, we release our Helm charts under the `paradedb` name.

We also ship our individual extensions (pg_analytics and pg_bm25) pre-compiled as .deb under our GitHub Releases, and as a fully complete product, ParadeDB, as a Dockerfile and Helm chart

philippemnoel··on PostgreSQL is enough
This is exactly why we built ParadeDB :)
philippemnoel··on PostgreSQL is enough
Citus is indeed an example for "distributed PostgreS". There are also serverless Postgres (Neon, Nile, AWS Aurora) which do this.

If you are interested in partitioning in an OLAP scenario, this will soon be coming to pg_analytics, and some other Postgres OLAP providers like Timescale offer it already

philippemnoel··on PostgreSQL is enough
I'm one of the makers of ParadeDB, a modern alternative to Elasticsearch. We build Postgres extensions to do fast search (pg_bm25) and analytics (pg_analytics). I love Postgres. If you have a small workload, like a startup, it certainly makes sense to stay within Postgres as long as you can.

The problem is, at scale, Postgres isn't the answer to everything. Each of the workloads one can put in Postgres start to grow into very specific requirements, you need to isolate systems to get independent scaling and resilience, etc. At this point, you need a stack of specialized solutions for each requirement, and that's where Postgres starts to no longer be enough.

There is a movement to build a Postgres version of most components on the stack (we are a part of it), and that might be a world where you can use Postgres at scale for everything. But really, each solution becomes quite a bit more than Postgres, and I doubt there will be a Postgres-based solution for every component of the stack.

← PreviousPage 5 of 6Next →