Hacker News Reader

I've seen a few good example blog posts discussing this scale, the 10 to 500 transactions per second and learned a lot from all of them.

Does anyone know of any description of architectures scaling to the next magnitude? 1000 to 10 000 TPS ? And even higher to say 100 000 to 1 000 000 TPS ?

View on HN

Its difficult to "guess" ahead of the fact how you are going to scale for the next 10x, because you need to know.

a.) Which component is going to start breaking, and that depends on the usage pattern.

b.) Which business/tech compromises are ok to make, and that depends on a.)

Generally speaking though, you'd try to benchmark the system to find the bottleneck component, and based on the nature of it, either try to throw more hardware (horizontal or vertical scaling) or optimize the software.

We handle peaks of 800k tps in few systems. It is for an analytical platform. Partition in Kafka by some evenly distributed key, Create simple apps that read from a partition and process it, commit offset. Avoid communication between process/threads. Repartition using kafka only. For some cases we had to implement sampling wherein usecase required highly skewed partitions.

Reply on news.ycombinator.com