armon

We cover why this is not done in the article, namely location transparency (our files aren't always in S3), and partial-failures. A simple client-side uploader like this doesn't handle cases like the upload failing part way through. For example, the user closing the browser or losing connectivity.

Full disclosure, I'm a HashiCorp employee, but another system to look at is Consul: http://www.consul.io. It is compared to ZooKeeper more extensively here: http://www.consul.io/intro/vs/zookeeper.html.

At a high level, ZooKeeper gives you a low-level distributed primitives. While these are all you need to build leader election, service discovery, monitoring, etc, it does require you to actually build that all yourself.

Consul goes the route of providing high-level features instead of low-level primitives (although they are all there, and usable via an HTTP API). It makes it easy as an operations person to deploy, and just as easy as an app developer to integrate and use.

My previous job was at an advertising firm, and we used HyperLogLogs for almost all of our real-time analytics infrastructure. They are incredibly space and time efficient. Each "counter" fits into about a single page of memory, and can count into the trillions with <2% error.

We developed an extremely high performance server around it (hlld): https://github.com/armon/hlld.

We were typically hitting with with tens of thousands of requests per second across about 50K counters. Although it was benchmarked to >1MM ops a second.

Similarly, we also make bloomd, which is an equivalent for using bloom filters, which provide a more set-like abstraction: https://github.com/armon/bloomd

Unless you develop natively on Linux, Docker must still run inside of a VM, so there is no real battery savings.

I think that Synapse already provides a plugin architecture. I t should be trivial to use the Consul HTTP API as a Synapse plugin. All this does is bridge the information Consul has and setup an HAProxy instance locally. If you want to go that route, should be pretty simple!

The two offer very different trade offs and have different architectures. Serf is an AP system, meaning it trades of consistency for availability. 90% of your nodes can be down and a Serf cluster will continue to operate. It does offer a primitive set of features, but it is useful for a wide variety of tasks such as configuring a memcache pool, load balancers, P2P VPN topologies, etc.

Consul is a CP system, meaning it trades availability for consistency. It has a much more limited ability to tolerate failures. However, its more central architecture allows it to support a richer feature set.

By keeping the tools separate we give developers and operators two different tools. Sometimes you need a hammer, and sometimes a screwdriver will do.

This page compares the two: http://www.consul.io/intro/vs/serf.html

To reiterate a bit on what hardwaresofton has already said, yes you could use Cassandra. However, Cassandra like ZooKeeper is just a building block. It doesn't actually provide service discovery, health checking, etc. You could use it to build all of those, but it doesn't provide it.

We compare Consul to ZooKeeper here, but much of that applies to Cassandra as well: http://www.consul.io/intro/vs/zookeeper.html

Internally, Consul could also use something like Cassandra to store the data. However we use LMDB which is an embedded database to avoid an expensive context switch out of the process to serve requests with lower latency and higher throughput.

SkyDNS is compared to Consul here: http://www.consul.io/intro/vs/skydns.html

That can probably be worded better. Serf operates at a node level abstraction, while Consul operates at a service level. In fact, Serf is used in Consul to power the discovery of other nodes.

You are right, the Consul site is very heavily based on the Serf site. It's actually almost entirely just a re-skin of the same site. The design is a bit cleaner however, which does make it nicer to read.

The value is base64 encoded, since JSON doesn't play nicely with binary values

We didn't manage to finish it in time, but it should be released in the next two weeks or so!

By this measure Memcached offers a greater feature set, an infinitely more robust parser, request concurrency, and doesn't really on a garbage collector. The very fact that this "db" relies on HTTP requests already makes me cringe.

Serf doesn't actually use Docker in anyway, just as clarification. It does work really well within it, but there is no dependency either way.

There were many potentially other solutions to this issue. The clients could have buffered writes locally, and done batch updates to MongoDB instead of incrementing each time.

Using a local SQLite, I can do "update counts set val=val+1 where id=?;" ~250K times per second on a Macbook Air. It would've been simpler to just do a local aggregate and then a batch update. That avoids Redis durability issues, or a secondary merge process.

FWIW, a local Postgres can do only about 3200 increments per second.

I've also worked in university research group settings, as well as a few startups in SV. I'd say there is a much stronger sense of the "whole" in these smaller groups, where everybody feels personally responsible for the success of projects. The smaller groups also take on the dynamics of the members, and the more eccentric personalities shape the culture and make the group a more fun and interesting place. You think of coworkers as friends first, coworkers second.

At Amazon, you don't get any of that. It is a very dry culture. It's clear that everybody is a coworker first. Eccentricity is largely looked down on (after all you need to worry about reviews and promotions, etc). Instead you get a sort of blah corporate culture, everything is grey, you talk about frugality as if somehow that's great. Rarely do you find super motivated engineers. Most people are fairly jaded, and aren't that committed to the success of projects. It's just a 9 to 5.

I've worked at Amazon, both in the warehouse management division as well as AWS, and the whole "bar raiser" phenomenon is mostly just marketing fluff. There are a handful of engineers who don't mind interviewing (or at least hate it less than most), and they get picked often to be part of the interview circuit. After some number of interviews, you are considered one of the mystical "bar raisers". However, in my experience, they are not any different from other interviewers. More experienced sure, but the difficulty of questions is highly variable, and honestly there isn't much culture at Amazon to worry about culture fit.

We built a similar decorator/context manager for profiling, but it is really much more useful if you can export the data to statsite + graphite so that you can graph and view it on an on going basis. The insight into the runtime is much more valuable when you have historic data to compare it to.

Riak Core provides a superset of the features of Serf. Riak Core uses gossip to manage membership, but it also provides quorums for coordination, and is based around the notion of a hash ring and virtual nodes. You could instead using Serf to build riak core like technology on top.

I think we are maybe just working with different definitions. High Availability for Serf means that it can continue to handle changes in topology and deliver user events in the face of node failures and network problems. However, it is inevitable that there will be a degradation in it's performance given network failures. If there are serious packet loss issues, Serf will mark a node as failed.

I'm not saying it "won't work well". It works as it is designed to. It will be available for operations, it will automatically heal when the partition recovers, and the state will be resynchronized with the "failed" nodes. The system will be in an eventually consistent state, which is expressly documented and is it's normal mode of operation.

If you consider 5% packet loss "minimal", I'm not sure what applications you are running. TCP degrades at over 0.1% packet loss, and most UDP streaming protocols have serious degradation over 5%.

You could build something like this on Serf, but it would take a little creativity. You might do something like this:

1) Send a "pre-deploy" event. Handler scripts use random number generator to decide which group they are in ("flip a coin" basically)

2) Half the nodes should transition to the "left" state, do the deploy and rejoin the cluster.

3) Once this is done, trigger the migration.

4) Flip the LB to the nodes that did the Join/Leave (you can potentially distinguish them using different role names, or by tracking who left and joined)

5) Run the "post-deploy". The other half of the nodes should now deploy

6) Update the LB to include everybody as the nodes leave/join

This is of course a rough sketch, it is certainly possible if tricky to build something like this.

In writing that section, we tried to provide "just the facts". If there is anything that seems wrong or misleading in any way, we'd like to know so that the page can be corrected. It is not our intention to say "Serf is good, ZooKeeper is bad". They are very different tools, and we are just trying to highlight the differences. In fact, we believe that the strongest use cases involve using those tools together.

I'd highly recommend taking a look at this page: http://www.serfdom.io/docs/internals/gossip.html. One of the great attributes of the gossip protocol is it is very robust to intermittent network failures. Under minimal packet loss conditions (<5%), the rate of false positives should be very low. This is due to a few techniques, one of which is indirect probing, and another is a novel "suspicion" mechanism. In the case of a network partition, the parts of the cluster can run in isolation and will recover when the partition heals. If you are interested, the paper referenced there ("SWIM: Scalable Weakly-consistent Infection-style Process Group Membership Protocol"), is the foundation of Serf. In the paper you can find more details about the behavior of the cluster, false positive rates under packet loss, and partition handling.

tl;dr the systems is in fact designed with network errors in mind, as opposed to handling them being an afterthought.

Dynamic roles is a fairly hard problem, which we hope to solve by building a different tool on top of Serf in the near future. If you want multiple roles that are static, it can be simulated by just providing a comma separated value as the role (since that is just an opaque string value to Serf)

Yes, the availability of the system is not tied to any given node(s). Any node (or group of nodes) can continue to operate in the face of failure.

The Serf executable is actually just a wrapper around the `serf` library. That library is designed to be embedded in Go applications. Documentation for the library is available here: http://godoc.org/github.com/hashicorp/serf/serf

We designed the `serf` library to be able to be easily embedded, so hopefully it can be of some use. Unfortunately Serf does not make use of any sort of NAT traversal currently. We've open sourced the project hoping to get the community involved, and NAT traversal is something we'd gladly work with the community to get implemented.

We have a comparison against Chef and Puppet, which may be relevant to Salt here: http://www.serfdom.io/intro/vs-chef-puppet.html. Not sure about Salt's search, but Serf is designed to run much more often than config management tools, and is able to handle topology changes in seconds instead of minutes or hours. Serf is also designed ground up to be fault tolerant, which is not usually a design goal of config management tools.

This might be a little unclear, but if you check the documentation for agent configuration (http://www.serfdom.io/docs/agent/options.html), there is an option to provide a role. The role is the metadata support currently

I'm guessing the authors deployed to production with no indexes on their collections. This means the cost of most queries was probably O(n). This could go unnoticed in development where N is small, but is unacceptable once you have any real data. I think they are recommending thinking about and adding indexes while you are developing as opposed to trying to do so post-hoc, once queries are taking tens of seconds.