Hacker News Reader

Comment by mozumder | Hacker News Reader

It seems like setting up and syncing an entire separate database for search wouldn't exactly be "trivial"...

I've migrated last week 500M documents (100GB) of data from mongodb to ES. It's quite simple and straightforward. Used the bulk API. Just make sure to have a good machine (16vCPU) and use parallelization and just wait. after 24hours, it just works.

That's only 200 bytes per document.

Can they be really considered 'documents' or are they actually database fields?

That's an average.

How do keep the ES index consistent with MongoDB?

Elasticsearch can be installed in seconds with most package managers and syncing is trivial depending on what you want to index. You can use a JDBC river for bulk indexing or rely on trigger's.

"Syncing is trivial" Have you got any links to help me set up PostgreSQL to Elasticsearch sync?

I don't know about elasticsearch specifically, but here are postgres -> x instructions using LISTEN/NOTIFY.

https://www.chrisstucchio.com/blog/2013/postgres_external_tr...

If it's so trivial, why don't you just put the instructions here so the rest of us can learn from your experience?

Rivers were deprecated and removed from 2.0. The syncing story isn't terrible but it's not as simple as it used to be.

that's disappointing :-(

Looks like the new syncing story involves logstash as an ETL platform

https://www.elastic.co/guide/en/logstash/current/deploying-a...

Eh, that's more for logs specifically. The "syncing story" is something that depends on your technology platform and specific situation. I adopted ES when it had rivers, but never even considered using them for syncing.

For what it's worth Logstash is cited as a replacement for rivers in the deprecation announcement.

https://www.elastic.co/blog/deprecating-rivers

> syncing is trivial depending on what you want to index

Can you explain how you keep your Elasticsearch index consistent with your main database, in face of system failure (for example a network partition or a machine hard reboot)? This is usually the non-trivial part.

Reply on news.ycombinator.com