Hacker News Reader

Comment by animalnewbie | Hacker News Reader

Is there a non-Java alternative to this ES/Logstash stuff? Preferably rust or a native lang, but okay with CLR too. I'm not comfortable running Java in production after previous memory issues...

View on HN

You can check out https://vector.dev/ to replace Logstash. Not sure about replacing Elasticsearch with something non-Java. Especially for the search use case -- Lucene is fairly dominant. For metrics you have prometheus (Go, not sure if that is better for memory issues with the non-tunable GC). You will probably want/need a clustered storage backend for prometheus. For that you have lots of options: https://prometheus.io/docs/operating/integrations/#remote-en... . Of those, TiKV (Rust), InfluxDB (Go), and TimescaleDB (C - its a Postgresql extension) seem like decent options.

This is obviously a dumb question, being in this thread and all but what's the difference between LogStash, Elastic Search and Lucene? I only ever saw one amazing demo once with real time search and I remember hearing all 3 names. My guess is LogStash stores and the other two retrieve?

AFAIK, Elasticsearch is the database system. Apache Lucene is the engine that Elasticsearch uses to perform searches, and the Logstash is the system that retrieves (or receives) data, processes it and gets it into ES.

Lucene is the core search engine, it can be used on its own and has a lot of depth in its many capabilities.

ElasticSearch (and SolrCloud >> LucidWorks Fusion Server) add a distributed architecture that leverages Lucene's capabilities.

LogStash helps ES easily deal with log files, and it has been one of the main marketing drivers used by elastic which is ironic since most of Lucene's capabilities are more useful on human generated text as opposed to machine generated logs.

Logstash accepts events from different types of sources, processes them (there are tons of inputs, filters and outputs available), converts them to Elasticsearch-compatible format and sends them to the cluster.

It is usually used as a central component.

For logs and metrics reading, one can use Elastic Beats,such as Filebeat for logfiles. Beats are lightweight agents written in Golang.

Recent versions of Elasticsearch also allow to perform the processing step directly in Elasticsearch, using ingest nodes and pipelines (my experience with it wasn't great). In that case Logstash isn't needed.

Not only is Go's GC non-tunable there's also no way to set a heap limit, so you have to rely on the oom-killer for memory limits.

You should give it another try. New versions of Java have made great progress in the memory area.

I feel like a lot of hate Java gets nowadays is largely due to historical reasons.

Though come to think of it, if you're talking about ES itself and not Java, then I have no idea, I never used ES in prod.

> New versions of Java have made great progress in the memory area.

Elastic uses huge amounts of memory (30GB is the norm). That's not a problem by itself, but Java garbage collections at this size can often take 10 seconds or more to complete on modern hardware. During this time, the server is basically down, the only solution is to increase timeouts or use replication servers.

> but Java garbage collections at this size can often take 10 seconds or more to complete on modern hardware.

For heaps of this size you can use Shenandoah GC and you get pauses well below 100ms. Your post is a perfect example of the type of 'historical reasons' (or historic FUD) the GP talks about.

> For heaps of this size you can use Shenandoah GC and you get pauses well below 100ms. Your post is a perfect example of the type of 'historical reasons' (or historic FUD) the GP talks about.

This GC is relatively new, and Elasticsearch doesn't support it (https://discuss.elastic.co/t/support-for-shenandoah-gc/16237...)

It may be a potential solution, but it's hardly a proven one. The concern is not FUD, it's legitimate.

Couchbase could be a reasonable alternative, depending upon your requirements. Couchbase is written in Erlang iirc and Couchbase's indexing is written in Go.

MarkLogic is an alternative, but it isn't free.

Reply on news.ycombinator.com