HNHacker News
TopNewBestAskShowJobs

philippemnoel

1,017 karma · joined February 19, 2019

Find me at https://linktree.com/philippemnoel
submissionscomments
philippemnoel··on We made Postgres writes faster, but it broke replication
The value prop for customers vs Elasticsearch are:

- ACID w/ JOINs

- Real-time indexing under UPDATE-heavy workloads. Instacart wrote about this, they had to move away from Elasticsearch during COVID because of this problem: https://tech.instacart.com/how-instacart-built-a-modern-sear...

Beyond these two benefits, then the added benefits are:

- Infrastructure simplification (no need for ETL)

- Lower costs

Speaking the wire protocol is nice, but it's not worth much.

philippemnoel··on We made Postgres writes faster, but it broke replication
Bear with me, this will be a bit of a longer answer. Today, there are two topologies under which people deploy ParadeDB.

- <some managed Postgres service> + ParadeDB. Frequently, customers already use a managed Postgres (e.g. AWS RDS) and want ParadeDB. In that world, they maintain their managed Postgres service and deploy a Kubernetes cluster running ParadeDB on the side, with one primary instance and some number of replicas. The AWS RDS primary sends data to the ParadeDB primary via logical replication. You can see a diagram here: https://docs.paradedb.com/deploy/byoc

In this topology, the OLTP and search/OLAP workloads are fully isolated from each other. You have two clusters, but you don't need a third-party ETL service since they're both "just Postgres".

- <self-hosted Postgres> + ParadeDB. Some customers, typically larger ones, prefer to self-host Postgres and want to install our Postgres extension directly. The extension is installed in their primary Postgres, and the CREATE INDEX commands must be issued on the primary; however, they may route reads only to a subset of the read replicas in their cluster.

In this topology, all writes could be directed to the primary, all OLTP read queries could be routed to a pool of read replicas, and all search/OLAP queries could be directed to another subset of replicas.

Both are completely reasonable approaches and depend on the workload. Hope this helps :)

philippemnoel··on We made Postgres writes faster, but it broke replication
You don't need to. Customers usually deploy us on a standalone replica(s) on their Postgres cluster. If a query were to take it down, it would only take down the replica(s) dedicated to ParadeDB, leaving the primary and all other read replicas dedicated to OLTP safe.
philippemnoel··on We made Postgres writes faster, but it broke replication
Our customers typically deploy ParadeDB in a primary-replicas topology, with one primary Postgres node and 2 or more read replicas, depending on read volume. Queries are executed on a single node today, yes.

We have plans to eventually support distributed queries.

philippemnoel··on We made Postgres writes faster, but it broke replication
Yes, Figma!
philippemnoel··on We made Postgres writes faster, but it broke replication
One of the ParadeDB maintainers here -- Being PostgreSQL wire protocol compatible is very different from being built inside Postgres on top of the Postgres pages, which is what ParadeDB does. You still need the "T" in ETL, e.g. transforming data from your source into the format of the sink (in your example CrateDB). This is where ETL costs and brittleness come into play.

You can read more about it here: https://www.paradedb.com/blog/block_storage_part_one

philippemnoel··on Sirius: A GPU-native SQL engine
> Postgres is not typically considered to "scale well," but oftentimes this is a statement about its tablespaces more than anything; it has foreign data[4] API, which is how you extend Postgres as single point-of-consumption, foregoing some transactional guarantees in the process. This is how pg_analytics[5] brings DuckDB to Postgres, or how Steampipe[6] similarly exposes many Cloud and SaaS applications. Depending on where you stand on this, the so-called alternative SQL engines may seem like moving in the wrong direction. Shrug.

Maintainer of pg_analytics (now part of pg_search) here. I 100% agree that the statements against Postgres are often exaggerated. In practice, we see both the smallest and the largest companies "just use Postgres" while mid-scale companies often overthink their solution.

That said, there are indeed phenomenal "alternate" SQL engines. I've seen many users see great success on tools like ClickHouse, which ParadeDB is not yet competitive with, and sometimes (dare I say) even Elasticsearch. As for whether this one is one of them... That I couldn't say

philippemnoel··on BM25 in PostgreSQL
Hi folks, ParadeDB author here. We had benchmarks, but they were super outdated. We just made new ones, and will soon make a biiiig announcement with big new benchmarks. You can see some existing benchmarks vs Lucene here: https://www.paradedb.com/blog/case_study_alibaba

This comparison isn't super fair -- ParadeDB does not have compatibility issues with Postgres and rather is directly integrated into Postgres block storage, query planner, and query executor

philippemnoel··on Apache DataFusion
That's true. We have some more ideas for DataFusion in the works, though... Stay tuned!
philippemnoel··on Datadog acquires Quickwit
Acquisitions don't necessarily mean the end of innovation. Sometimes, it allows them to take innovations they've worked hard on for years and expand the reach to a significantly larger audience :)

I have met the founders of all 3 of these companies and can assure you they all care tremendously about bringing their work to the world.

ParadeDB is independent and without plans to sell anytime soon, though :)

philippemnoel··on Datadog acquires Quickwit
pg_search dev here -- Thanks for mentioning us.

Re: Tantivy. I'm hopeful the community Paul and the Quickwit team have built on top of Tantivy will continue to flourish. I'm sure Datadog will build product(s) with Quickwit, which is built on Tantivy and will contribute to it. Many other companies like ours (ParadeDB) and other databases also integrate it. I can't speak for others, but we'll contribute whenever possible. We're currently working on supporting nested documents in Tantivy, for example, and hoping to upstream this work.

While it's reasonable to be concerned, I'd say this is a win for Quickwit, Tantivy and, of course, the well-deserving team behind them.

philippemnoel··on Show HN: SeekStorm – open-source sub-millisecond search in Rust
ParadeDB (paradedb.com) is similar to this HN, but baked into Postgres to solve this very problem you are describing
philippemnoel··on Pg_analytics is now PostgreSQL Licensed
New York is very much alive in the software world :)
philippemnoel··on Pg_analytics has switched to PostgreSQL license
I know right! We worked with this individual designer from Reddit. If you want their name, shoot me an email/linkedin/etc. and I'll connect you. He was great to work with.
philippemnoel··on Pg_analytics is now PostgreSQL Licensed
In some ways :). We've come to learn this feature should be truly open to the community, and are doing what we can to provide options to people. We had already done the work, so might as well make it accessible to as many as possible! :)
philippemnoel··on Vector databases are the wrong abstraction
We could add support for something like `pg_vectorize` in order to generate embeddings directly from the database. We simply haven't seen enough demand yet. Perhaps we haven't listened hard enough :')
philippemnoel··on Vector databases are the wrong abstraction
BM25 implemented in Postgres as a Postgres extension: https://github.com/paradedb/paradedb (disclaimer: I work for ParadeDB)
philippemnoel··on Everything I built with Claude Artifacts this week
Cool to see ParadeDB is now in Claude/ChatGPT :)
philippemnoel··on MVCC – the part of PostgreSQL we hate the most (2023)
The whole ParadeDB team is really excited for OrioleDB and Supabase to ship this :) It's long overdue in the Postgres ecosystem!
philippemnoel··on Elasticsearch is open source, again
This is huge. We wrote "Why we picked AGPL" only a few weeks ago, discussing why ParadeDB (an Elastic competitor!) chose the AGPL. Glad to see Elastic is joining forces on the AGPL front.
philippemnoel··on We can and should domesticate raccoons
I want one. Where do I sign?
philippemnoel··on Postgres as a Search Engine
I would recommend you give ParadeDB pg_search a look. It's got support for tokenizers in all major languages. You can think of it as the next generation tsvector for more complex workloads.

Disclaimer: I work at ParadeDB

philippemnoel··on Postgres as a Search Engine
So happy to hear you like it. We have a lot more coming up for it. Stay tuned!
philippemnoel··on Postgres as a Search Engine
This is partially true. You're right that the feature set is necessary, but it's now possible to do this in Postgres.

ParadeDB pg_search bundles Tantivy, a Lucene-inspired library inside Postgres, to give users the feature set (BM25 ranking, tokenizers, faceted search, etc.) while keeping the benefits of Postgres (keep your data normalized, avoid ETL, etc.)

Disclaimer: I work on ParadeDB

philippemnoel··on Postgres as a Search Engine
Keeping it in Postgres avoids needing to denormalize the data, which is a big pain point. ParadeDB pg_search is based on Tantivy, a Lucene-inspired library, so in some ways it allows you to both benefit from a Lucene-based system and also keep it in Postgres
philippemnoel··on Postgres as a Search Engine
ParadeDB pg_search supports tokenizers in all major languages
philippemnoel··on Transformers for Ruby
ParadeDB founder here -- We migrated the pg_analytics project to its own repo and will be adding JSON support!
philippemnoel··on Why we picked AGPL
That's so cool that he responded! Thank you for sharing
philippemnoel··on pg_duckdb: Splicing Duck and Elephant DNA
ParadeDB itself is AGPL, yes. Our core offering is pg_search, which offers Elasticsearch inside Postgres. What we build will be AGPL, and if pg_duckdb moves forward we will be happy to rebuild our analytics offering on top of it.
philippemnoel··on pg_duckdb: Splicing Duck and Elephant DNA
ParadeDB here -- agreed. The syntax for ParadeDB and Crunchy Analytics is similar, since we're both built as Postgres foreign data wrappers. pg_duckdb is the odd one out here, being built as a custom scan. I am confident they will converge in the future, though :)
← PreviousPage 2 of 6Next →