Hacker News Reader

We wanted to use the KCL on the receiver side, but figured that it required building some sort of cluster to mediate which node would receive which shard. Tricky, especially when resharding. With Lambda, you get that for free. So we opted for lambda, but weren't overly happy. Did you solve that problem?

View on HN

The KCL itself handles shard assignment and balancing. Each application has its own dynamodb table where it stores lease information. If you have 64 shards and want that processed on one machine? Spin up one instance of the application. The KCL will spin up 64 sub processes.

Instead want to spread the load? Spin up as many instances of the application you want, they don't need to be clustered, they will negotiate, using the lease table to take and balance the shards across all instances.

I'll add that the same goes for resharding. The KCL sees and handles the Parent/child shard relationship, getting a lease on the children, but waiting until the parent is "drained" before reading from the children. And it handles this through the shenanigans that occur when resharding in non-multiple/evenly-divisible numbers

Reply on news.ycombinator.com