Hacker News Reader

Comment by illumin8 | Hacker News Reader

I'm not a GlusterFS expert, and haven't used it before, but you should know that most consensus algorithms (Paxos, Raft, etc) only function reliably with an odd number of nodes. I have to wonder if your problems were mostly self-inflicted from having 2 nodes. Of course, any network partition in a 2-node cluster has a huge potential for data corruption, as each node now thinks it is the master (split-brain).

In a 3-node cluster, any system with a decent consensus algorithm (to be clear, I'm not sure if GlusterFS has one) would know that during a partition the cluster can only continue to operate if at least 2 nodes can communicate with each other to elect a new master.

View on HN

I didn't do the initial configuration myself but I wouldn't rule out this kind of gross, self-inflicted problem.

One has to wonder why the software even lets you run it with an unsafe number of nodes though... (Lots of distributed software is guilty of this...)

Uh, well if you base your GlusterFS experience on that, no wonder you have a bad view on the technology. I can only assume that your setup only used two nodes, which is not really supported, and of course will cause split-brain problems. And two bricks, really? GlusterFS is best aimed for scale both on the server and client side, two bricks is not really scale.

There's probably a good case study to be made here on why so many people have had bad experiences with GlusterFS. If the problem is that it's too easy to set it up in a fragile way, then they need to fix that, because otherwise you're going to end up with a lot unhappy users (as you can see in the other comments).

Still, if I can't make it work well on a reasonably small scale, why would I expect it to work any better when I scale it up?

I would suggest reading the documentation and understanding the architecture. It works well for many applications but some are less ideal. Maybe yours was a corner case? I don't know. But it's used by many, for instance Facebook (https://www.socallinuxexpo.org/scale/14x/presentations/scali...).

> Of course, any network partition in a 2-node cluster has a huge potential for data corruption, as each node now thinks it is the master (split-brain).

This is not a given, the cluster can and should refuse to operate if it can not get a majority vote for the master, if that is required to prevent data corruption. With two nodes that means both nodes must be active and reachable, with three nodes one node can fail.

But then in a split with a 3-node cluster, there is now a 2-node cluster in charge, and what happens if another partition happens?

It depends on the underlying algorithm. These risks are bounded, not solved.

Typically, clusters without a majority (2 out of 3, 3 out of 5, 4 out of 7, etc) of nodes present will shut themselves down to prevent data corruption.

Reply on news.ycombinator.com