Hacker News Reader

Comment by willvarfar | Hacker News Reader

(My memory is that we kept on influx including 1.7, but it was a while ago now so memory might be fuzzy)

I guess influx perf and efficiency is really depending a lot on your data shape then :)

Our experience was that performance dropped off a cliff if you had too much data, too much tagset cardinality, or else your query was too broad. And when it failed, it lost data.

In fact, it lost data generally. When we were replacing it we dual-ran an ACID DB version (which, with tokudb, was fast enough to keep up (although we didn't index every tag column)). So we did a diff and discovered just small random holes in the influx data that we'd never noticed before.

We had other considerations when we went mysql, as in, we were already using it. If shopping for a standalone solution to start a new project on, I'm thinking Timescale is the go-to these days?

View on HN

I’ve been self-hosting InfluxDB in the hundreds of GB range for several years. I wouldn’t say I’m super happy with it, but… let’s say we’ve reached an understanding, the software and I. We’re on the latest patch of 1.8 and content to stay there.

I agree with GP about storage efficiency, which is superb. Query performance is good as long as a single query doesn’t deal with more than ~dozens of series. And $deity help you if you want to do hourly roll-ups of all series for a short time range, as RAM usage is wildly unpredictable. Storage is optimized for long reads of a single series, not for short reads of many series (but in fairness, you have to choose one or the other, that’s just the physics of the thing).

If I were starting from scratch, I’d probably pick Timescale. Or maybe DuckDB… I wonder if it would work for our use case.

> And $deity help you if you want to do hourly roll-ups of all series for a short time range, as RAM usage is wildly unpredictable.

I think I went properly mad while trying to troubleshoot this. The same query sometimes pulls 5GB, sometimes 20GB, sometimes 50GB and sometimes OOMs at 200GB memory pulled beyond base load of the system. And there's no query planner, no execution log, no metrics to help you. And most documentation or threads about it can be summarized as "well sucks to be you, eh? Maybe less data would be an option I guess"

We don't do that anymore and just roll up a very small select number of metrics.

And yeah, we've committed to Postgres as our main DB 2 years ago or so, and currently time is clearing up to start work with TimescaleDB. Zabbix is supposed to be great with it.

I have pretty much exactly the same experience.

However, I do feel that they are trying to really do the right thing with the new 3.0 architecture, addressing the deficiencies (most importantly performance and full-fledged SQL) while keeping the stuff that works (InfluxQL for simple and legacy queries). Also leveraging open-source projects and contributing to their upstream is a plus. Thus I’m hoping for them to succeed delivering on that promise.

Agreed, embracing battle-hardened open source tech will be a win for them and for customers.

However, once your storage layer is parquet and your query layer is SQL, well... DuckDB is also basically parquet+SQL, and it won't be long before there's a nice Postgres wire protocol adapter in front of it. What's the advantage of continuing to use InfluxDB if you don't need clustering or HA?

Reply on news.ycombinator.com