Hacker News Reader

Comment by bjclark | Hacker News Reader

It's not a big deal when you do it the first time. It's a big deal when you need to start adding servers. What do you do?

View on HN

I don't have scaling problems, I don't have sites that big, yet.

He's pointing out a situation where it becomes a bigger deal, to answer your question. While there are solutions in practice for this situation, it can be painful.

I thought he wanted to know what I did. Since I already said I don't know much about this problem domain, I'd be curious why it's a problem when you start adding servers.

One such bad situation would be if you have numeric keys that you shard based on a mod 2. You do this because you want to distribute data between 2 servers, so one will get all even keys and the other will get the odd keys.

A few months down the line your load gets so high that you need to add a third server, but your sharding mechanism is based on mod 2 - so you need to hack around this in one of many ways to utilize the third server effectively.

Make sense?

So don't people just publish client libs that handle the details for you? It doesn't seem like an unsurmountable problem, but once again, I've never dealt with it in practice.

So don't people just publish client libs that handle the details for you?

Sometimes, but oftentimes sharding issues are unique enough that they require custom solutions when the backend technology does not support it natively.

And these sharding issues don't happen with the things he marks as "scales!"?

That is correct. Things like S3 can expand without issue, and others like Voldemort handle partitioning data natively.

MySQL Cluster http://www.mysql.com/products/database/cluster/

Reply on news.ycombinator.com