Announcing UltraWarm (Preview) for Amazon Elasticsearch Service
aws.amazon.com
aws.amazon.com
Some orgs just need a simple ES-compatible system that won’t lose data. No frills. I inherited an e-commerce contract that uses ES and maintaining a cluster has been more costly than it should be, even with t2 nodes. The demand on this isn’t high but it needs a few nodes to protect the data.
Most people just run one node. It works fine until the node falls over and comes back with zero indexes.
That's completely unacceptable for any non-toy workloads. If you want a single node, deploy something like PG. You aren't using any of the "Elastic" capabilities, so why bother?
If the volume of data is that small, it's better to run 3 small nodes than a single, larger one.
For any non-trivial cluster, you'll want 30GB of RAM for Elasticsearch(no more), plus whatever you can spare as OS cache. 64GB is common. Plus a lot of storage, preferably on fast disks.
CPU consumption really depends on what you are doing but both ingestion/indexing and searches are costly. Ingestion is mostly I/O bound, but still requires a surprising amount of CPU.
Since we are talking about hot-warm, retention is the problem. You'll need enough nodes with capacity to hold all the required data. Every open index uses some amount of memory for each shard. A shard should ideally not exceed 50GB, so if you have big indices you'll have lots of shards (or you have lots of small indices but still lots of shards).
You could have really enormous disks, but you shouldn't. If there is a problem with any given node (and at scale, there will be), now your cluster will want to move ridiculous amounts of data if it detects a node failure. You can mitigate this, but you don't really want to turn of shard relocation.
Now multiply the storage requirements by two, as I assume you'll be using at least one replica per shard.
So now you have a bunch of nodes doing ingestion, and a bunch of other nodes storing data and answering search queries (in many clusters, they are the same, but you can specialize). It's not uncommon to end up with hundreds of nodes. At this scale, network communication becomes a bottleneck too, as the cluster is trying to send enormous state updates. And now masters will have trouble keeping up, so you'll end up scaling them as well.
It's a snowball. Small ES clusters are trivial to manage, large ones are a full time job, even with automation. If this solution can help offload data from the primary ingest or hot cluster, it will be a godsend.
By sticking to shards that are 10-100GB, and only using i3 instances, you can get great performance. My company ingests about 2 TB per day and we use less than 10 i3 instances.
AWS' key innovation seems to be that hot nodes are the ones you provision to keep both primary and secondary indices for hot data (writable and recent), and ultra-warm nodes, of different ec2-instance-type than the hot nodes, prefetch speculatively / predictively caches of index-data downloaded on-demand from S3 required to execute read-only requests transparently to the rest of the cluster.
Impressive: It is almost like running ultrawarm nodes with Amazon FSx for Lustre instead of EBS.
I wonder if the Lucene / Elasticsearch changes required to do this will make it to OpenDistro. If not, that'd be tremendously disappointing [1].
---
[0] https://www.elastic.co/blog/implementing-hot-warm-cold-in-el...
[1] In addition, the innovation focus has shifted from furthering the open source distribution to making the proprietary distribution popular. This means that the majority of new Elasticsearch users are now, in fact, running proprietary software. https://news.ycombinator.com/item?id=19359602
I've faced that pain on AWS ES service and will definitely give it a try.
Interesting how well AWS is adding features on top of open-source in competition with the original vendor now.