Sector/Sphere is significantly faster than Hadoop
sector.sourceforge.net
sector.sourceforge.net
The most interesting thing about Sector/Sphere is that it doesn't use TCP/IP for transport but their own UDT stuff, which has its own pluses (more efficient on high latency links) and minuses (stability issues due to lack of wide usage/testing)
The really interesting question is how Sector/Sphere scales and how well it deals with outages.
But that does not say whether the 'read' or 'write' that failed will be restarted or whether it starts the whole segment from scratch.
Which is what's interesting in Hadoop, it apparently (I haven't tried that) keeps chugging along is if literally nothing happened, so from the applications point of view the fault never occurred.
I wonder how much of a tax it takes to report the result of each operation sequentially vs. doing it in a batch at the end of the segment. If the difference in performance is because Hadoop reports sequential while sector does it batched, be nice to have something that gives you a sliding bar to chose somewhere along the gradient between failover optimization and optimistic optimization where your failovers are far and few in between.
This is the same problem with Haskell/Parallelism with functional data structures. Maybe you can write a linearly scaling parallel quicksort on linked lists Haskell, but I can write an array based C sort that's 1000x faster. You could say that that's unfair and use arrays in Haskell, but now parallelism isn't so easy anymore.
The only time that I tried Hadoop I was so underwhelmed with the performance that I immediately decided that it wasn't for us. For a while glusterfs looked like it might be a winner but I'm getting more and more skeptical over time.
Pohmelfs (in the linux kernel) is an absolute pain to get working at all. So anything that will give us that cluster file system with good failover will get a look, if performance is acceptable compared to what we do today. (custom built CDN).