At a previous company, we dealt with huge data streams (~1TB data / minute) and our customers expected real-time aggregations.
Making an in-house solution for this was incredibly difficult because each customer's data differed wildly. For example:
- Customer A's shards might have so much cardinality where memory becomes an issue.
- Customer B's shards might have so much throughput where CPU becomes a constraint. Sometimes a single aggregation may have so much throughput where you need to artificially increase the cardinality and aggregate the aggregations!
This makes the optimal sharding strategy very complex. Ideally, you want to bin-pack memory-constrained aggregations with CPU-constrained aggregations. In my opinion, the ideal approach involves detecting the cardinality of each shard and bin-packing them.