Ask HN: How do you handle time-series metrics at scale without a large bill? | Hacker News Reader

Ask HN: How do you handle time-series metrics at scale without a large bill?

4 pointsxpbl3 comments

I want to build my own first party analytics for fun and profit and I'm trying to find a good timeseries database that has a sweet spot between costs, overhead, and scalability.

I've considered just standard PostgreSQL, SQLite on the edge (Cloudflare D1 / Turso), purpose built columnar databases (Clickhouse, DuckDB) and the ones managed by big cloud like Google BigQuery, Amazon Timestream, and Cloudflare Analytics Engine.

For those running event analytics on indie / small-team budgets: what has your actual production experience been? Trying to make an informed decision.

View on HN

we did something similar, raw events go to clickhouse on a cheap vm and rollups hourly/daily, dashboards only read the rollups. bigquery got expensive fast for us once dashboards started refreshing alot

for the live stuff (visitors right now etc) i just keep counters in memory and flush every few seconds

we had similar concerns and issues and that's why started building an in memory db with time series support built in, mostly for that live part. happy to share if your intrested in giving it a try. I know this is in-memory, different from the disk based traditional systems you would be looking for, but if it would cost less and perform better for your use case then it be a good shot

I recommend append-only & rollups if it's not a crazy scale. I've seen it handle 250k events daily. Plan your indexes, if you miss on add it concurrently.

We’re on azure and a surprisingly good choice has been to use their eventhub to ingest the telemtry and then to consume it in batches of a few seconds using a consumer. Time series is hard because it is a lot of data that comes in all the time, this allows you to relax those requirements a bit.

After eventhub we’re using ducklake. But I’m not at the point where I’d flat out advise it to other people (just code immaturity that tk their credit is being worked on)

Also, if you’re evaluating postgres, check out timescaledb.

Reply on news.ycombinator.com