Hacker News Reader

Comment by speedgoose | Hacker News Reader

Have you tried column based databases such as Cassandra, HPE vertica, or Clickhouse?

Have you tried a "big data" approach like Apache Spark on parquet files?

Did you consider saving less data? Do you really need to save that much data? Can't you sample the data and save a lot less while keeping the same information overall ?

View on HN

We are yet to try any other technology and this post is primarily for seeking those candidates.

We could not get away with saving less data, for reasons beyond the scope of this post (non-technical reasons).

I would try first with these columns based databases but I think you should consider saving less data. Doing what you want is definitely possible but it may be expensive, and if it's only because one non technical co-worker would love to query so much data for his reports, perhaps you should try to show him the cost once you tried a few technologies.

Querying less data but saving everything is also an alternative. With Clickhouse you can specify a sampling rate for example.

This turned out to be a good advice. I am evaluating if it is possible to somehow intelligently sample the data when it comes to ELK. Something like an average of 1min data via a logstash filter. We could do this from our backend programs too, but for now, I am trying to do this in ELK during the writes.

Also we do not use Clickhouse, but I will see if I can somehow do a sampling rate in the reads, during my read queries. Thanks.

Reply on news.ycombinator.com