Hacker News Reader

Comment by sysexit | Hacker News Reader

I love CoreOS and they've done some super impressive engineering. But really, a new storage system? Rewriting in Go and using etcd for central state management makes things easier, but this is still a hard problem.

Some things that need to be solved sooner or later: data replication so that N faults of X entities are protected against (X can be disks, enclosures, racks, data centers, regions, ..), recovery from failed disks, scrubbing, data management, backups, some kind of storage orchestration and centralized management.

If you look at Ceph which IMHO represents the state of the art in software defined storage, it took many years to get it to a point that it was usable. I hate to be cynical but in this case I would be surprised if CoreOS can pull this of. I wish them all the best though and would be happy for them if I am wrong.

View on HN

> this is still a hard problem

Which is why I'm rather surprised that these systems continue to be released without any formal specification or model checking. It's hard to get these things right... worth the effort, IMO, to create formal specifications.

> But really, a new storage system?

I think the post outlines quite well why they chose to build a new distributed storage layer. From a product perspective it makes a lot of sense for them to have an offering in this space. Also, the fact that it's a hard problem is what makes it a compelling product offering. You can't make much money solving easy problems.

> Also, the fact that it's a hard problem is what makes it a compelling product offering. You can't make much money solving easy problems.

If they can solve the hard problem, yes. Which I believe is unlikely.

It's like any time we take a step forward in one area we have to reinvent the last 50 years of computing to support it.

Persistent storage is a "hard problem" Really?

Persistent storage isn't a hard problem. Distributed, well performing, scalable and consistent storage is a hard problem.

Actually, persistent storage is fairly hard in itself. Look at what ZFS does to ensure data integrity in the face of phantom writes, dropped writes, bad controllers, and other implicit, non-fatal failures.

ZFS still also have the issue of having to perform well. You have a point, but ZFS is still trivial compared to a proper distributed filesystem, and you could achieve the same reliability much easier than ZFS if you sacrificed the performance.

The ClusterHQ guys behind the FlockerHQ found this out the hard way [0]. Initially Flocker was meant to provide a container data migration tool on top of ZFS, now it is a front-end to more established storage systems like Cinder,vSan,EBS,GPD and so on.

[0] https://docs.clusterhq.com/en/latest/faq/#what-happened-to-z...

Absolutely -- I didn't mean to imply that ZFS even comes close to solving the distributed parts of the problem, but rather that a distributed storage system does have to address the problems of putting bits on disk.

> Actually, persistent storage is fairly hard in itself

Don't we have distributed data storages precisely because it's impossible to guaranty persistence locally? It's kind of a way to not bother trying to solve the impossible, but to achieve some guarantees on a different level.

That's one use case, but far more common is reducing latency and increasing bandwidth for distributed computing acting on a shared set of data.

Persistent storage on today's filesystems and complex hardware is a hard problem. All kinds of failures can happen during any write. Some are obvious with some silently corrupting data. There's been decades of work on approaches to dealing with this with a variety of tradeoffs. Picking the right one for a widely-deployed, portable, distributed app is tricky by itself.

They're aiming to do a lot more than that. ;)

I've been trying to find performance evaluations of distributed file systems, and in most tests I've seen Ceph is a lot slower than alternatives like GlusterFS.

When you say Ceph is "state of the art in software defined storage", are you including performance, or are there advantages in features or reliability where you think it outclasses the competition?

Reply on news.ycombinator.com