Hacker News Reader

Comment by klaussilveira | Hacker News Reader

This doesn't seem to be buffering inserts to ClickHouse: https://github.com/Openpanel-dev/openpanel/blob/c90848765a6a...

So any major website will have problems with it. I would suggest moving this into a stream (NATS, Kafka) or just having some sort of service that buffers events, then offloads to Clickhouse in batch.

View on HN

Correct! Currently no batching but no need to make things complicated for now.

Can extend the current redis queue to batch request or clickhouse batch engine.

Looking forward to have these problems hehe.

Please DM if you want to chat about best approach!

Or enable async inserts on the ClickHouse client, which won't require changing to the buffer table engine.

https://clickhouse.com/docs/en/optimize/asynchronous-inserts

Have read about this, but totally forgot about it! Will try it out!

Won't this be a problem as soon as a website reaches, say, 600 RPS? Which is not much.

Dunno, that’s already a fairly big website. The one I was working on in the past with 15k customers did around 15RPS. The large enterprise system I work on with 3k users does slightly less than that. That covers an awfully large amount of use cases already, and then you can get 30x more?

I have time to grow. Don’t think I’ll hit this level just yet.

As someone using clickhouse, for similar event ingestion stuff - I’d lean towards implementing it sooner rather than later.

It’s not too hard, and it can surprising _how much more_ Clickhouse likes bulk queries.

Sure thing! Will look into it more during the coming weeks. Feels like I have the most of the dashboard complete now.

But I really want to start giving the native app some love as well

This is the correct answer! any early project needs to make big trade-offs and premature optimisation has killed many projects. You'll never please everyone (especially on HN) but you're not making it for everyone

Reply on news.ycombinator.com