Hacker News Reader

Comment by brown9-2 | Hacker News Reader

Interestingly the cause of the most severe part of their downtime seems to still be unknown to them:

As a next step, we introduced a new shard, intending to move some of the data from the overloaded shard to this new one.

We wanted to move this data in the background while the site remained up. For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go down.

To those who use MongoDB - does this sound like something that might have been caused by MongoDB itself, or Foursquare's use of it?

View on HN

Without knowing more about their schema, it's hard to speculate. However, let's assume they had a single, large "checkins" collection, sharded by user ID as they said. That would be distributed across the shards. If the mongod process is down on one shard and therefore the shard isn't responding, the data just wouldn't be available.

But it sounds like the shard was still up. As per the sharding FAQ[1]:

"What if a shard is down or slow and I do a query? If a shard is down, the query will return an error. If a shard is responding slowly, mongos will wait for it. You won't get partial results."

They said they had a performance issue on the overloaded shard so perhaps it's possible that mongo believed the shard to be up, when it was instead overloaded. This meant any queries just waited, taking down the whole site.

As I said, this is speculation and as a massive production user of MongoDB ourselves[2], I'm interested to know more.

[1] http://www.mongodb.org/display/DOCS/Sharding+FAQ [2] http://blog.boxedice.com/2010/02/28/notes-from-a-production-...

But it wasn't the overloaded shard that brought the site down (it was only slowing things down), it was the addition of the new shard:

"For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go down."

If the data was being moved it could cause even more load on an already overloaded shard.

"What if a shard is down or slow and I do a query? If a shard is down, the query will return an error. If a shard is responding slowly, mongos will wait for it. You won't get partial results."

an approach to CAP that chooses consistency over availability. I wonder why as it isn't a bank and they could have chosen the availability instead.

If in doubt, choose consistency.

I know nothing about MongoDB but why would one shard that's responding slowly affect queries to all the other shards and take the whole site down? I can't believe any database system would be designed to work like that.

At 6:30pm EST, we determined the most effective course of action was to re-index the shard, which would address the memory fragmentation and usage issues.

Sounds similar to some of the problems we're facing with MongoDB too. Indexes fragment a lot, especially if you delete documents frequently. You have to reindex everything eventually.

MongoDB offers background indexing (except for _id fields!) but it's not really an option if you need the indexes at all times to keep executing queries. Also, you'll need to reindex _id eventually and there's currently no way to do that without blocking the server.

I'm also wondering why this kind of intervention is necessary at all. The NoSQL solution we use at work has load based automatic splitting, and I'd have thought (though I haven't confirmed) that this would be an obvious feature to include.

Which NoSQL store do you use?

http://labs.google.com/papers/bigtable.html

That looks potentially useful in some cases. Can you send me the code/docs for that?

http://hbase.apache.org/

Does it work that well for production for you? Really, that's extremely interesting.

Also look at dynamo based systems like cassandra and riak (riak seems to have a better load balancing at the moment, cassandra is a bit more "bumpy")

Load based splitting is on the MongoDB roadmap, but doesn't exist yet.

Just because the feature exists, doesn't mean it works.

I would speculate that it's a poorly-chosen shard key. MongoDB's built-in sharding uses range-based indexing. If you choose user_id as your shard key, and those are autoincrementing integers, then you're screwed if newer users tend to be more active on average than older ones.

Wait, people shard on a key other than something approximately random, like an sha1 hash!?

Where I work (Etsy) we keep an index server that maps each user to a shard on an individual basis. There are a number of advantages to it. For example, if one user generated a ton of activity they could in theory be moved to their own server. Approximately random works for the initial assignment.

Flickr works the same way (not by coincidence, since we have several former Flickr engineers on staff).

Sounds like a good system. I've noticed that people tend to do things like shard based on even/odd, and then they realize that they need three databases.

I've never had either problem though... but if I ever need to shard I plan on doing it based on object ID. Then one request can be handled by multiple databases, "for free", increasing both throughput and response time.

Even/odd isn't the end of the world, but you would then be best jumping to mod 4.

Actually for anonymous sharding (without a central index) a consistent hash is about the closest you can get to ideal distribution and flexibility. I haven't looked but I presume that's what mongo uses under the hood for their auto-sharding, too.

> To those who use MongoDB - does this sound like something that might have been caused by MongoDB itself, or Foursquare's use of it?

MongoDB still has problems with locking too much for certain operations. The 10gen guys keep on introducing "yields" to those operations, but the simple (and fast) database/process-level locking could very well cause this... (not a 100% 'yes', but it seems possible)

Reply on news.ycombinator.com