HNHacker News
TopNewBestAskShowJobs

philippemnoel

1,022 karma · joined February 19, 2019

Find me at https://linktree.com/philippemnoel
submissionscomments
philippemnoel··on pg_duckdb: Splicing Duck and Elephant DNA
It seems like the team behind it is now working on pg_duckdb, though, which likely will supersede Citus columnar eventually? Is that not the goal with it?
philippemnoel··on pg_duckdb: Splicing Duck and Elephant DNA
There is a lot missing from it, as you know. We'd be happy to be part of the project if we get commit access/even partnership :)
philippemnoel··on pg_duckdb: Splicing Duck and Elephant DNA
This is a cool roadmap. I'm excited for Neon to get in the mix here, it's going to be a lot of work. My concern is... There are very few companies who care about doing the work in such a Postgres-idiomatic way. I suspect Neon will have to be one of the main backers of such a large project for it to pick up.
philippemnoel··on pg_duckdb: Splicing Duck and Elephant DNA
Hey Jerry, it's nice to see you here :) Truth is, the work you guys did at Hydra with the fork of Citus columnar is the more Postgres-idiomatic analytics engine in Postgres. It's a shame it was abandoned.
philippemnoel··on pg_duckdb: Splicing Duck and Elephant DNA
As Craig said, Crunchy has a very enterprise mature offering for analytics in Postgres and are very much leading the charge here. ParadeDB is built in a similar way, also ranking high on ClickBench, and is available in the open-source as well.

I'm hopeful the pg_duckdb project will mature enough to be a stable foundation for ParadeDB and others, but that appears to be a matter of MotherDuck and how much they're willing to push this forward.

philippemnoel··on pg_duckdb: Splicing Duck and Elephant DNA
ParadeDB has an open PR for time_bucket support via DuckDB here: https://github.com/paradedb/pg_analytics/pull/32
philippemnoel··on Why we picked AGPL
We've had 35 contributors in total on ParadeDB thus far, with 5 of them being core team members.

That said, we measure traction in terms of users of the product rather than in terms of contributors to the project. We welcome contributions but understand that this is primarily a project led by our company and don't expect any external contribution.

philippemnoel··on Why we picked AGPL
While I agree with you that these are non-commercial, by now many of them have very big commercial backings. Many Postgres core contributors work at Microsoft, AWS, etc. who make billions of dollars selling Postgres-as-a-Service today.

Those projects aren't inherently commercial, and great software can be built by passionate hobbyists, but often times the two go hand-in-hand

philippemnoel··on Why we picked AGPL
The best way to avoid any such issue is to contribute changes to the upstream project, which is our goal with the choice of license. We're hoping for a centralized ParadeDB project from which everyone can benefit.
philippemnoel··on Why we picked AGPL
That's fair. We use AGPLv3. I haven't seen any AGPL project that wasn't using v3, so we decided to keep it short for readability, but you bring up a very good point.
philippemnoel··on Why we picked AGPL
Blog author here -- I couldn't have said it better. If you'd like to meaningfully contribute to the project, we're hiring :)
philippemnoel··on The new PostgreSQL 17 make dist
pg_analytics maker here -- That's cool! We package ours in CNPG here: https://github.com/paradedb/helm-charts

Would you recommend using Nix even in that context?

philippemnoel··on Why we picked AGPL
Blog author here. That's fair! We'll add horizontal scaling eventually, but for now, our customers (including some large brands!) have not needed it. Turns you single nodes are remarkably powerful nowadays
philippemnoel··on Why we picked AGPL
We have some API-compatibility in faceted search. But yes, we are not (today) a drop-in replacement for ES. We have many migrations from ES/OpenSearch and users find that converting their code over is well worth the savings in maintenance, no-ETL and increased uptime, though.

That said, we may one day add ES API compatibility if we can :)

philippemnoel··on Why we picked AGPL
We've looked into the DCO before. It turns out CLAs are more common and easier to deal with, thanks to a tool like cla-assistant.io. We ended up opting for it for the sake of simplicity.
philippemnoel··on Why we picked AGPL
Blog author here. I couldn't agree more. Thank you for your kind words and for sharing the academic perspective of using the AGPL - super interesting!
philippemnoel··on The future of F/OSS continues to be AGPL
Blog author here -- You're right, they can. We're aware of this. But, for most cloud vendors, they do need to modify the code in order to make that possible (if only to integrate it within their stack).

It then becomes a game of can you design your software in such a way that it will never be possible for cloud vendors to host it without modification, for example by keeping some required features closed-source (say a control plane, etc.).

For now, we feel this is good-enough protection and we hope that by doing our work right, we can give as much as possible to the community while still keeping guarded just enough to prevent cloud vendors from eating our lunch.

philippemnoel··on Full text search over Postgres: Elasticsearch vs. alternatives
ParadeDB supports partial indexes as well: https://docs.paradedb.com/search/full-text/index#partial-bm2...
philippemnoel··on Full text search over Postgres: Elasticsearch vs. alternatives
ParadeDB founder here. We are open-source: https://github.com/paradedb/paradedb. Very much not "invite only". We have hundreds of open-source deployments in production and several paying customers.

Our bring-your-own-cloud solution, which is our primary hosted service, is indeed in developer preview and if anyone is interested in using it, you can contact us. It will enable adding ParadeDB to AWS RDS/Aurora.

philippemnoel··on Full text search over Postgres: Elasticsearch vs. alternatives
Manticore is primarily oriented towards the MySQL world
philippemnoel··on Full text search over Postgres: Elasticsearch vs. alternatives
It adds a small overhead to transactions. The exact number depends on how much you're inserting and indexing, but is in the milliseconds.

We have adopted strong consistency because we've observed most of our customers run ParadeDB pg_search in a separate instance from their primary Postgres, to tune the instance differently and pick more optimized hardware. The data is sent from the primary Postgres to the ParadeDB instance via logical replication, which is Zero-ETL. In this model, there are no transactions powering the app executed against the ParadeDB instance, only search queries, so this is a non-issue.

We had previously adopted weak consistency because we expected our customers to run pg_search in their primary Postgres database, but observed this is not what they wanted. This is specifically for our mid-market/enterprise customers. We may bring back weak consistency as an optional feature eventually, as it would enable faster ingestion.

philippemnoel··on Full text search over Postgres: Elasticsearch vs. alternatives
Agreed! But for most of our customers, they want to separate their OLTP Postgres from their Search Postgres. This is because query patterns differ, and you want to tune Postgres and choose hardware differently. This also provides higher reliability and ensures search queries can't slow down transactions.

ParadeDB implements this by ingesting logical replication (WALs) from the primary Postgres. This gives a "closely coupled" solution, where you have the feel of a single Postgres database (ParadeDB is essentially an extra read replica in your Postgres HA setup). More here: https://docs.paradedb.com/replication/pg_search

philippemnoel··on Full text search over Postgres: Elasticsearch vs. alternatives
Correct. If you run your own Postgres, you can install pg_search directly, otherwise we suggest our users integrate with their existing Postgres deployments (say AWS RDS) via logical replication: https://docs.paradedb.com/replication/pg_search
philippemnoel··on Full text search over Postgres: Elasticsearch vs. alternatives
This is a very good explanation of what we've found keeps users on Elastic: The combination of FTS and fast facets/aggregates in the same query. At ParadeDB, we've implemented support this in what we call aggregations: https://docs.paradedb.com/search/full-text/aggregations
philippemnoel··on Full text search over Postgres: Elasticsearch vs. alternatives
RUM indices are a nice improvements, but they're not in Postgres core and likely won't ever be :( If you're looking for Elastic-level FTS in Postgres, your best bet is likely to overengineer pgvector+tsvector or simply to use ParadeDB pg_search
philippemnoel··on Full text search over Postgres: Elasticsearch vs. alternatives
Arango is sweet! We've actually talked to many people who switched from Postgres to Arango because of its better FTS. This was one of the reasons for creating ParadeDB in the first place. Many of the users who made this switch wished they could have stayed in Postgres without compromising on FTS.
philippemnoel··on Building and scaling Notion's data lake
This is one of the best blog posts I've seen that showcase the UPDATE-heavy, "surface data lakes data to users" type of workload.

At ParadeDB, we're seeing more and more users want to maintain the Postgres interface while offloading data to S3 for cost and scalability reasons, which was the main reason behind the creation of pg_lakehouse.

philippemnoel··on Dense Vector and Sparse Vector and Fulltext and Tensor Reranker = Best for RAG?
Hey folks, ParadeDB co-founder here. Cool project! Just thought I'd chime in and clarify a few things:

1. pg_sparse is deprecated. pgvector released native sparse vector support with the `sparsevec` datatype, and ParadeDB no longer maintains pg_sparse. It has been this way for several months already.

I'd love to see a benchmark re: Tantivy. You claim that pg_search is much slower, but Tantivy is state-of-the-art for full-text search performance and the ParadeDB performance is robust. You can see our benchmarks in our repository README, where we compare ourselves to Elastic.

4/5. ParadeDB is Postgres by design. If you are adopting Postgres, which many are, then ParadeDB can be installed directly as an extension via logical replication on a read replica. This removes the need for ETL to a non-Postgres system, which drastically reduces operational burden.

Of course, if you're not using Postgres, ParadeDB is not designed for you and a tool like Infinity seems like a viable option alongside other standalone search engines.

philippemnoel··on Dense Vector and Sparse Vector and Fulltext and Tensor Reranker = Best for RAG?
BM25 is indeed way more important than these vector DBs will claim. At ParadeDB, we've observed significant use cases where customers need both
philippemnoel··on Putting DuckDB in Postgres to Query Iceberg
What is your use case? If you're considering pg_analytics, I would suggest you check out pg_lakehouse. It's even more performant than pg_analytics, see here: https://github.com/ClickHouse/ClickBench/pull/208
← PreviousPage 3 of 6Next →