1,022 karma · joined February 19, 2019
I'm hopeful the pg_duckdb project will mature enough to be a stable foundation for ParadeDB and others, but that appears to be a matter of MotherDuck and how much they're willing to push this forward.
That said, we measure traction in terms of users of the product rather than in terms of contributors to the project. We welcome contributions but understand that this is primarily a project led by our company and don't expect any external contribution.
Those projects aren't inherently commercial, and great software can be built by passionate hobbyists, but often times the two go hand-in-hand
Would you recommend using Nix even in that context?
That said, we may one day add ES API compatibility if we can :)
It then becomes a game of can you design your software in such a way that it will never be possible for cloud vendors to host it without modification, for example by keeping some required features closed-source (say a control plane, etc.).
For now, we feel this is good-enough protection and we hope that by doing our work right, we can give as much as possible to the community while still keeping guarded just enough to prevent cloud vendors from eating our lunch.
Our bring-your-own-cloud solution, which is our primary hosted service, is indeed in developer preview and if anyone is interested in using it, you can contact us. It will enable adding ParadeDB to AWS RDS/Aurora.
We have adopted strong consistency because we've observed most of our customers run ParadeDB pg_search in a separate instance from their primary Postgres, to tune the instance differently and pick more optimized hardware. The data is sent from the primary Postgres to the ParadeDB instance via logical replication, which is Zero-ETL. In this model, there are no transactions powering the app executed against the ParadeDB instance, only search queries, so this is a non-issue.
We had previously adopted weak consistency because we expected our customers to run pg_search in their primary Postgres database, but observed this is not what they wanted. This is specifically for our mid-market/enterprise customers. We may bring back weak consistency as an optional feature eventually, as it would enable faster ingestion.
ParadeDB implements this by ingesting logical replication (WALs) from the primary Postgres. This gives a "closely coupled" solution, where you have the feel of a single Postgres database (ParadeDB is essentially an extra read replica in your Postgres HA setup). More here: https://docs.paradedb.com/replication/pg_search
At ParadeDB, we're seeing more and more users want to maintain the Postgres interface while offloading data to S3 for cost and scalability reasons, which was the main reason behind the creation of pg_lakehouse.
1. pg_sparse is deprecated. pgvector released native sparse vector support with the `sparsevec` datatype, and ParadeDB no longer maintains pg_sparse. It has been this way for several months already.
I'd love to see a benchmark re: Tantivy. You claim that pg_search is much slower, but Tantivy is state-of-the-art for full-text search performance and the ParadeDB performance is robust. You can see our benchmarks in our repository README, where we compare ourselves to Elastic.
4/5. ParadeDB is Postgres by design. If you are adopting Postgres, which many are, then ParadeDB can be installed directly as an extension via logical replication on a read replica. This removes the need for ETL to a non-Postgres system, which drastically reduces operational burden.
Of course, if you're not using Postgres, ParadeDB is not designed for you and a tool like Infinity seems like a viable option alongside other standalone search engines.