HNHacker News
TopNewBestAskShowJobs

philippemnoel

1,037 karma · joined February 19, 2019

Find me at https://linktree.com/philippemnoel
submissionscomments
philippemnoel··on PostgreSQL is enough
This is exactly why we built ParadeDB :)
philippemnoel··on PostgreSQL is enough
Citus is indeed an example for "distributed PostgreS". There are also serverless Postgres (Neon, Nile, AWS Aurora) which do this.

If you are interested in partitioning in an OLAP scenario, this will soon be coming to pg_analytics, and some other Postgres OLAP providers like Timescale offer it already

philippemnoel··on PostgreSQL is enough
I'm one of the makers of ParadeDB, a modern alternative to Elasticsearch. We build Postgres extensions to do fast search (pg_bm25) and analytics (pg_analytics). I love Postgres. If you have a small workload, like a startup, it certainly makes sense to stay within Postgres as long as you can.

The problem is, at scale, Postgres isn't the answer to everything. Each of the workloads one can put in Postgres start to grow into very specific requirements, you need to isolate systems to get independent scaling and resilience, etc. At this point, you need a stack of specialized solutions for each requirement, and that's where Postgres starts to no longer be enough.

There is a movement to build a Postgres version of most components on the stack (we are a part of it), and that might be a world where you can use Postgres at scale for everything. But really, each solution becomes quite a bit more than Postgres, and I doubt there will be a Postgres-based solution for every component of the stack.

philippemnoel··on PostgreSQL is enough
There is now https://github.com/paradedb/paradedb/tree/dev/pg_analytics (disclaimer: I am one of the makers of pg_analytics)
philippemnoel··on Multi-database support in DuckDB
DuckDB is in-memory. It makes sense as a in-memory columnar cache. We went with DataFusion to benefit from the deep integration it has into the ecosystem with Arrow, Parquet, etc.

As for Delta Lake, we would've loved to do Iceberg but the Rust crate for Iceberg is very new, and doesn't have enough functionalities yet. We're tracking the project and planning to contribute, so that we can add support for Iceberg eventually

philippemnoel··on Ask HN: What tool do you use to query your data lake?
Basically just data dumped in S3 buckets with Iceberg or Delta Lake formatting :shrug:
philippemnoel··on We built our customer data warehouse all on Postgres
One of the authors of ParadeDB here. While ad hoc aggregations in vanilla Postgres are slow due to lack of column-oriented storage, extensions like pg_analytics are addressing that problem. In our view, one of the main use cases of a data warehouse is when you need separation of storage/compute, which enables distributed analytics and scalable storage. With Postgres, query processing + storage are all happening on the same node. That being said we're looking at ways to separate out storage to an external data lake as part of pg_analytics.
philippemnoel··on Multi-database support in DuckDB
Check out https://github.com/paradedb/paradedb/tree/dev/pg_analytics, we're shipping this week for column-based storage in Postgres. It's not LSM, though, but Parquet/Arrow
philippemnoel··on Multi-database support in DuckDB
Hey! I'm one of the makers of pg_analytics/ParadeDB.

We store tables column-based. You can do CREATE TABLE (...) USING deltalake; when using ParadeDB and it will be stored column-based (and soon, with support for also querying cloud object stores!)

philippemnoel··on ParadeDB – PostgreSQL for Search
This is interesting. We've been seeing a pretty split desire where some companies would prefer ParadeDB installed as an extension in their existing database, while others would prefer ParadeDB as a separate search node (for independent scaling, recovery, performance, etc.)

We expect production-scale ParadeDB to be used as a separate search node getting logical replication from your primary Postgres. This would give you easy-to-maintain and scale services while benefiting from isolation of services (so your search can't take down your primary database, etc.)

We would be open to what you describe and happy to work it out. But we are currently thinking of prioritizing separate hosting of ParadeDB. It's early, though, and still unconfirmed what customers will prefer :)

philippemnoel··on Thoughts on PostgreSQL in 2024
This is a great post. As an extension developer myself (pg_bm25), seeing more native support for search-related functionalities that extensions can harness would be really great.

I'm surprised nothing was mentioned around pgBouncer/connection pooling. Having more native support for general connection pooling has been a requested feature in Postgres for a while, and many popular DBMS have it built-in. Just how some of what Patroni offer should be better bundled in (as the author argues), I think the same could/should be true for pgBouncer

philippemnoel··on Thoughts on PostgreSQL in 2024
I think bringing RocksDB into Postgres as a Table access method would really be a powerful feature, as an alternative to heap
philippemnoel··on ParadeDB – PostgreSQL for Search
Hi there! ParadeDB author here.

We don't believe AGPL should prevent you from self-hosting for your own internal usage. We see ourselves as becoming a Postgres-based Elasticsearch alternative, primarily used internally for observability, etc.

If you are interested in using ParadeDB to power your own product, or would prefer a different license, we'd love to hear from you. You can reach me directly at phil@paradedb.com.

We spent a lot of time evaluating which license to use, and decided to follow the footsteps of Citus Data and go with a standard OSS license while ensuring that we can't be hosted on public clouds without a partnership. Hope this helps!

philippemnoel··on ParadeDB – PostgreSQL for Search
ParadeDB author here. You're right, currently we only do the "Search" in Elastic, but we're planning to do real-time analytics, logs processing, etc. eventually
philippemnoel··on Fly Postgres, Managed by Supabase
ParadeDB developer here -- we're developing a way for users to no zero-ETL sync between any managed Postgres and ParadeDB, so that you can get a dedicated search node on Postgres without slowing down your production DB. We expect this will be the primary way in which people use ParadeDB in the future.

That being said, we understand that for startups it might be easier to have pg_bm25 directly within their production database, especially if data volume is small. We're actively open to working with other Postgres-as-a-Service providers in the space to make this possible :)

philippemnoel··on Show HN: I scraped 25M Shopify products to build a search engine
You can even do Elastic-level full text search in Postgres with pg_bm25 (disclaimer: I am one of the makers of pg_bm25). Postgres truly rules, agree on the rec :)
philippemnoel··on OpenAI's board has fired Sam Altman
I just got an email saying they're moving to pre-paid billing... Seems like Sam Altman might've hidden some very large financial costs that the board just discovered?
philippemnoel··on Pg_bm25: Elastic-Quality Full Text Search Inside Postgres
ParadeDB and pg_bm25 offer weak consistency. pg_bm25 doesn't slow down transactions for indexing, and like ElasticSearch it becomes become eventually consistent shortly after (typically at most a few seconds, altough your mileage may vary based on the amount of data modified in the transaction(s)).
philippemnoel··on Pg_bm25: Elastic-Quality Full Text Search Inside Postgres
Hey! Absolutely, we would love to offer as many cloud providers as possible for our compute backend. We're starting with AWS, and will be adding other clouds based on demand. I've added Scaleway to our list, and if you'd like to help us bring ParadeDB to Scaleway we would love to work together to make it happen faster.

In the meantime, you can self-host ParadeDB on Scaleway directly by running the Docker container. Hope this helps!

philippemnoel··on Pg_bm25: Elastic-Quality Full Text Search Inside Postgres
Our goal is for one day ParadeDB to be a viable alternative to AWS RDS/Aurora, so that like you say, you don't need to keep data in-sync and can just use one system (ParadeDB). Soon it will be possible for you to have ParadeDB running on your AWS (utilizing your cloud credits+all security/privacy guarantees) but be managed via the ParadeDB dashboard, similar to how Aurora works from a developer UX.

Of course if you are 100% attached to AWS RDS itself (rather than the convenience of AWS RDS, which is replicable by ParadeDB), then there's not much we can do here, as we also need to eat :')

philippemnoel··on Pg_bm25: Elastic-Quality Full Text Search Inside Postgres
Yes! We have another extension, pg_search, which is specifically for hybrid search using pg_bm25+pgvector. You can find it here: https://github.com/paradedb/paradedb/tree/dev/pg_search
philippemnoel··on Pg_bm25: Elastic-Quality Full Text Search Inside Postgres
pg_bm25/ParadeDB author here. What we're doing is building an opinionated alternative within PostgreSQL. If you are not using Postgres, or want your system to be separate, Elastic is still the best choice and will likely remain so.

Other people have brought up great points for why or why not to switch. Our vision for this is that ParadeDB is not merely "better" than Elastic, but rather different. Elastic will never be a PostgreSQL database, and we'll never be a NoSQL search engine. If you want one or the other, you'll pick either ParadeDB or Elastic.

philippemnoel··on Pg_bm25: Elastic-Quality Full Text Search Inside Postgres
Thanks for sharing, will look to add a benchmark for that as well
philippemnoel··on Pg_bm25: Elastic-Quality Full Text Search Inside Postgres
pgrx is awesome and making pg_bm25 would've been infinitely more challenging without it. Check them out if you want to make a Postgres extension, we can't recommend them enough
philippemnoel··on Pg_bm25: Elastic-Quality Full Text Search Inside Postgres
That's right, we do support pgvector (it is pre-installed on ParadeDB) and support full HNSW. In fact, we even have another extension, called pg_search, which is the combination of searching on pgvector and pg_bm25 for better results! Topic of another blog post to come sometime soon :)
philippemnoel··on Pg_bm25: Elastic-Quality Full Text Search Inside Postgres
ParadeDB author here -- correct! We plan to offer a hosted version soon and the idea behind picking AGPL is to be as permissive as possible so that people can use the product for free, but also protect ourselves from abuse in case a large company, say AWS, were to want to ship it in their own environment.

In fact, we went through much questioning wondering to go with ELv2, Apache, AGPL, etc. before settling on AGPL

philippemnoel··on Pg_bm25: Elastic-Quality Full Text Search Inside Postgres
One of the ParadeDB authors here, hey! Thanks for pointing this out, you're completely right. That's an oversight on our end. We'll update the benchmarks and re-run them to correct this :)
philippemnoel··on Pg_bm25: Rust-based full text search engine in Postgres
Yes! That will be coming very soon :)
← PreviousPage 6 of 6