Hacker News Reader

Comment by frugalmail | Hacker News Reader

The canonical open source alternative to S3 https://wiki.openstack.org/wiki/Swift

Riak CS is another one:

Ran this in production and dealt with a lot of issues. I would caution people against it's use in anything critical or customer facing.

Could you please elaborate it? What were the issues you were facing?

As another user with nothing but negative experiences with Riak-CS in production, I thought I'd take a stab here. We had a 12-node cluster with ~10TB per node, fwiw. In no particular order:

- The restart times of the Riak process ranged from 10 minutes to 3+ hours, during which time the cluster was basically useless. Not a single suggestion from support sped up this process.

- Every single night from 0800 - 0900 UTC, the cluster would grind to a halt (as measured by canaries measuring upload/download cycle times). This continued even after we migrated all customer data and traffic off of the cluster.

- Riak-CS ships with garbage collection disabled despite it being a critical feature. I inherited a cluster that had been run for some months without gc enabled. Turning it on caused the cluster to catastrophically fail. Basho Support, over a period of close to a year, was unable to find a single solution that would get our cluster back to health. If our cluster were a house on a show like Hoarders, the garbage in it would be considered load bearing.

- We attempted to upgrade our way out of our un-garbage-collect-able mess, but the transfer crashed. Every. Single. Time.

- Even had transfers worked, all of the bloated manifests have to be copied in their entirety, so you can't gc the incoming data on the new cluster.

- Even while babying the cluster, it would become unusable at least once a month, requiring a restart of all nodes. The slowest node took 3+ hours to start, followed by another 3+ hours of transferring data. This was 6+ hours of system downtime every month.

- During these monthly episodes, we attempted to engage with support and try to debug the processes (we were a team of seasoned Erlang developers). We could attach Observer and/or use the REPL to grab stats, but not a single support resource was able or willing to engage.

- For giggles, once we had migrated all users off of the cluster, we attempted to let gc run. It never completed. Not once. We let this go on for a few months before nuking the entire cluster.

Now, I absolutely realize that we got ourselves into that mess by running the cluster without gc for an extended period. But in the grand scheme of things, this cluster wasn't storing a very large amount of data -- tens of TB spread over tens of millions of objects. Having the cluster get into a state where gc can never run and where this causes snowballing instability is unacceptable.

We switched to Ceph. We've never looked back.

We didn't have any issues with lost data but we had a lot of operational issues that didn't have clear fixes. Primarily around TLS, migrations, and performance. We had to contact support for many issues because the documentation for various failure modes wasn't there.

For what it's worth, I've worked with it for a couple of customers with pretty large Riak stores and never ran into or heard of any problems myself.

I've used the official JS aws-sdk and boto3 as clients.

We use riak and riak-cs in production and in fact we are one of biggest riak users.

If you use riak in production you probably do want their (Basho) support. Their product when works, works great, but when there is a problem it's a bit hard to troubleshoot it without knowing erlang and being familiar with riak's source code.

I use Swift at work, and while it is a great tool, it is a bitch to set up. I would be curious to learn how Minio works more technically on a distributed level: how is object replication handled? are downloads automatically routed to the closest server? can I make downloads temporarily available (think Swift tempURLs)?

We are currently working on the distributed version and will be making a beta release soon.

Currently minio supports

- pure FS backend with single disk - pure Erasure coded backend with multiple disks on single node (like ZFS)

For more information you can read here - https://docs.minio.io/docs/minio-erasure-code-quickstart-gui...

We do not do any sort of replication erasure code handles disk failures and we also implement transparent bit-rot protection as well.

To replicate one setup to many you can use 'mc mirror -w' which would watch on events and do continuous replication.

Relevant docs can be found here

https://docs.minio.io/docs/minio-client-complete-guide#mirro...

Additionally "SwiftTempURLs" equivalent is called PresignedURLs in S3 API so we indeed support that as well.

Relevant docs here https://docs.minio.io/docs/using-pre-signed-urls-to-download...

Also http://pithos.io/ backed by Cassandra

And another good example is ceph+radosgw.

I run OpenStack Swift at work, currently working on deploying it on Kubernetes. Swift is a very fine piece of software, with a pleasant operations experience, but it will take a lot of time to set up initially. Plan at least half a man-year until you have it all up and running.

Reply on news.ycombinator.com