Hacker News Reader

Comment by brigadier132 | Hacker News Reader

why do companies use databases like elastic search when they could just use inverted indexes in postgres?

Performance, features, and scalability

I love Postgres, but you're comparing apples and oranges. You don't have the same level of query sophistication neither the same indexing quality you get from eg. elasticsearch. As a quick example, the other day I got a bug in PostgreSQL because "searching" does not perform diacritics conversion. Also, stuff like stop words, similarity search and whatnot either required dedicated indexes or are downright "round hole square peg" problems.

I believe they're being rhetorical, and agree that there are good reasons to use elasticsearch over postgres.

I use Elasticsearch along with PostgreSQL, but I don't think of Elasticsearch as a database. It's an external index that I denormalize some of my data into.

ES is a good parallel to a vector database.

run a search against ES => hydrate records from postgres

run a search against a vector database => hydrate LLM context from documents

Neither is a primary canonical datastore. Postgres is for ES and S3 is for documents.

so, its just happened that there is no good implementation of full text search in postgres, but fundamentally nothing prevents from implementing such search, except horizontal scalability if it is needed.

the pg extension called "rum" is very performant on millions of pdfs.

https://github.com/postgrespro/rum

my case is tens of billions of short docs, and default search is fine except index creation is single threaded, so it takes multiple days to build index, and github page says rum is even slower..

Reply on news.ycombinator.com