What has been abysmal for you? Maybe your use case is more advanced than ours, which is mainly absorbing logs from all over the place and doing the typical dashboard and alerts on them (with Grafana).
What has been abysmal for you? Maybe your use case is more advanced than ours, which is mainly absorbing logs from all over the place and doing the typical dashboard and alerts on them (with Grafana).
This is why you haven't noticed any issues.
> Amazon Elasticsearch Service lets you store up to 3 PB of data in a single cluster, enabling you to run large log analytics workloads via a single Kibana interface.
"You're not the DoD so of course you're not having issues".
It can't be more than a minuscule fraction of Amazon ES customers that are getting anywhere close to 3PB.
It's insane. Unless you're literally Facebook, or ingesting data from CERN's LHC... what possible use case requires 100GB of text data ingest per day?
Maybe it's a case of someone throwing a Service Mesh into a Microservices K8s cluster and logging all the things?
90%+ of our logs are severity INFO or have no severity at all. It's like pulling teeth to even get devs to output logs using the corporate standard json-per-line format with mandatory fields.
Still, once you're running hundreds of VMs processing a big data pipeline it's not hard to end up with massive amounts of logs. It's not just logging, really, it's also metrics and trace information.
Normally that's no issue as the data is stored in SANs and not sent onto the cloud for analysis, just giving a perspective.