Comment by outworlder | Hacker News Reader

Comment by outworlder | Hacker News Reader

Parent Full thread

> Then now they figure out this log thing. ElasticSearch is great but try to run it yourself. On a 50 nodes k8s, I bet your first ES config will down after 10mins the first time FluentD come up and send all the log under the sun since the cluster come up

Tell me about it. I have not one, but multiple k8s clusters (some over 50 nodes), sending data to a single Elasticsearch cluster. A lot of tuning was done, and it's not yet perfect.

Elasticsearch is amazing, but it requires external support, and the tools available to do that are not on par. Curator specifically – it's dumb as a rock and is very limited on what it can do. Even a hot/warm architecture is a challenge to do on curator alone, you should ideally have custom scripts to manage it. Which is a shame as this is one of the "reference" architectures by Elastic. Whatever Curator does should really be part of ES itself.

The Kibana + ELK combo won't give you things like log tailing (logtrail is hackish and hard on the servers). The log forwarders are horrible to work with (be it logstash, filebeat – or worse: fluentd).

Some days it just works and you are happy. Some other days, you wonder why we moved away from syslog senders and text files...

I actually have same conclusion as you. ES isn't easy to tune and run well.

A thing many overlook are restart it. You cannot just go and `systemctl restart` it, once that occurs, constant rebalance happenning because it puts the load to the rest of system and eventually bring them down, all around :(. That make it harder to operator in K8S when it needs time to detach/attach volume, plus the overhead of overlay network.

So I myself didn't know about a perfect tunning. But the log search on Kibana is super helpful though.

That's why I kind of sold on this Loki thing. I have a good feeling that software in Go tend to require less config/tunning compare with Java. Eg, InfluxDB.

> Some other days, you wonder why we moved away from syslog senders and text files...

I wonder the same. I actually like to grep log with tail more than a crappy web ui like Kibana, Sumologic etc.

Tailing log in a webui is clunky.

> once [a reboot] occurs, constant rebalance happening because it puts the load to the rest of system and eventually bring them down, all around

I have this same problem with a 3 node cluster, no Kubernetes. This was using Elasticsearch 2.3 and it worked for a year or two and then all of a sudden, any network interruptions would cause the thing to get split brain and all indices would go red. It would take 8+ hours to go back to green and usually required a reboot of all servers or else it would never recover.

It was happening often enough that the decision was just run a single ES node since the total data was only 25GB :(

> Whatever Curator does should really be part of ES itself.

Just as an FYI, Index Lifecycle Management (ILM) is landing in 6.6 and provides some of the features that Curator supports: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/...

Basically, it allows you to define things like hot/warm/cold architecture, rollover, retention, etc at the index-level, inside Elasticsearch. Will be landing as beta

> Whatever Curator does should really be part of ES itself.

It is becoming part of ES. Index Life Cycle management UI is coming up in Kibana.

> The Kibana + ELK combo won't give you things like log tailing (logtrail is hackish and hard on the servers)

Recently, in 6.5.0 log tailing was added. Now one can look at streaming logs live.

> The log forwarders are horrible to work with (be it logstash, filebeat – or worse: fluentd)

I'd like to know the problem you faced.

>"The log forwarders are horrible to work with (be it logstash, filebeat – or worse: fluentd)."

Why are they horrible to work with? Why is fluentd worse than the other two?

Reply on news.ycombinator.com