Let's Create a Simple Load Balancer with Go
kasvith.github.io
kasvith.github.io
If you like this kind of thing, we are developing a very powerful and flexible reverse proxy with load balancing into Caddy 2: https://github.com/caddyserver/caddy/wiki/v2:-Documentation#...
It's mostly "done" actually. It's already looking really promising, especially considering that it can do things that other servers keep proprietary, if they do it at all (for example, NTLM proxying, or coordinated automation of TLS certs in a cluster).
If you want to get involved, now's a great time while we're still in beta! It's a fun project that the community is really coming together to help build.
edit: by way of example, I used to work for a service that customers would upload files to, that's all that the traffic was. There was wild variability in the size, processing cost, and upload speed of each request. None of the standard load-balancing approaches really balance "load" from a service perspective. While things worked, it was rarely optimal.
I think most proxies are planning to move logic like this to the control plane. Envoy's gRPC stuff has some ways to dynamically throttle traffic to backends.
Load balancers really need to become programming runtimes, imo. Config languages aren't very expressive, and almost everyone needs their own logic at the LB level.
I _just_ put together a demo of latency based load balancing using HAProxy + awk and it's neat, but still very rudimentary compared to what I could express in, say, JavaScript: https://github.com/superfly/multi-cloud-haproxy
I agree that load balancer a do need the expressiveness of programming languages, but ideally that would only be with some typing and ability to easily un it test.
I'm really hopeful about deno (https://github.com/denoland/deno) for this. TypeScript is nice, the deno TCP perf is good, all it needs is some good proxy libraries.
In some of our early testing on basic workloads, we found that it's up to 2x faster than NGINX+Lua, largely because it does not require a VM. (This is a broad generalization, and we need to specifically optimize for these cases -- but this approach holds promise.)
Both the load balancing policies and the circuit breakers are extensible, meaning it is easy to change their behavior and add new ones as needed.
I could also imagine a specific load balancing policy that adds up a cost for each request using headers such as transfer size; i.e. dynamic weights. This would be a great contribution to the project if you are interested!
Caddy 2 also has an embedded scripting language that can make this kind of logic scriptable and dynamic, but that's still a WIP.
Nice to see how simple it is now though. Go is definitely a great choice for low-level networking, and .NET Core has recently become a great option as well.
The memory management, type safety, and high-level productivity make it great for building load balancers and other infrastructure components.
1. https://www.ageofascent.com/2019/02/04/asp-net-core-saturati...
One nitpick is that the autho reversed the meaning of active and passive health checks. Active generates new traffic to the backends just to determine their healthiness, passive judges this based on the responses to normal traffic.
type Backend struct {
URL *url.URL
Alive bool
sync.RWMutex
ReverseProxy *httputil.ReverseProxy
}The mutex is only used inside of SetAlive() and isAlive(), they're the only things this need to handle locking and unlocking. You don't want anything external to that calling the methods on RWMutex.
I haven't read the code so I can't verify if that's the case here, but I read OPs post as being worried about method clobbering (which is really a non-issue, if the popularity of embedding mutexes shows us anything).
And yet we have only a handful and one of the most popular charges money for cool features and does not appear to have an ABI for addons.
Apache, nginx, haproxy, envoy, traefik, and probably dozens more that aren't on the top of my head.
Also there exists an http module for emacs: https://www.emacswiki.org/emacs/HttpServer
> > Apache, nginx, haproxy, envoy, traefik, and probably dozens more that aren't on the top of my head.
> You literally listed a handful ;)
How many solutions to a problem are needed when the problem is well-defined before there is no longer a need to grab for more?
And we've started creating them for other fields, like visual effects: https://landscape.aswf.io/
Several of those are ingress routers, and I’ve seen no claims of those being able to run standalone (ie without specifically k8tes) so they are technically load balancer, except they can’t operate standalone. Say I wanted to run one per server instead of NodeJS cluster mode. I could not substitute those, right?
One is F5 which nobody says anything nice about anymore, and was positioned as a hardware solution for most of its good years.
I’d also note that a number of these are pretty young, indicating that there was a power vacuum that is now filling in.
HAProxy on the other hand has been doing some really fantastic stuff lately that is in the open source. It makes me a bit sad that the former is the "go to" and HAProxy doesn't get the usage it deserves.
To your specific point though I think it's just super tricky to get all the features people expect at the performance they expect as well. You essentially need to implement an very efficient HTTP server, also apparently not trivial, in order to get expected Layer 7 features.. Embedded scripting language support is quickly becoming de facto.. Simple traffic proxy/forwarding is the easy part but getting something competitive together feature and performance wise feels way beyond teeth cutting?
EDIT: Using golang as an example. You would need to either piggy-back the best HTTP server available for your language platform or write your own. The stdlib Golang http server is not competitive performance wise. The reasons for this are pretty well known and seem to be largely accepted by the core team(AFAICT). This won't impact most app and service developers too much, but for an LB service the number of cycles it leaves on the table my not be acceptable at all.
Not meant to be my own criticism, just an observation. I believe the net/http interface ossified its own design decisions making it hard/impossible to be compatible with?
That's why fasthttp uses a different design.
2 channels needed, 1 for available backends and 1 for broken ones.
On incoming request, the front end selects an available backend from channel 1. On completion, the backend itself puts itself either back onto channel 1 on success, or channel 2 on error.
Channel 2 is periodically drained to test the previously failed backends to see if they're ready to go back onto channel 1.
Plus you know that if the backend is on the channel, you know it's ready to accept, and you don't need the alive flag with synchronized access.
Fewer mutexes isn't necessarily an advantage, there will be far higher contention on that single mutex. Channel writes also require a lock.
One to write, that's used for writing only, not both read/write as in the published design.
Edit: it's a variation on the following, but for ReverseProxies, not worker goroutines http://marcio.io/2015/07/handling-1-million-requests-per-min...
Channels appear to be magical, since they're baked into the language. But package sync APIs are generally faster.
I quite don't understand what this means? What race conditions? Can anybody explain? Thanks.
To do this, there are three operations, SET s.current to +1, and GET s.current so that it can be MODDED with servers.length.
If that SET and GET are not coordinated between threads, then a race could sour the index. For example, if two threads call this method at the same time, they could both SET the +1 before either GETs the value, then they will both get the same value +2 from where it started instead of +1 for each caller.
What do you mean? These seem synonymous to me.
> I thought Go has first class primitives to be able to do this?
TBH I'm only 1 week into learning Go myself (coming from Java/Scala/JS/etc). But it looks like the article used what Go offers. They used the atomic package which says, "provides low-level atomic memory primitives useful for implementing synchronization algorithms."
What I meant was similar to this talk: https://blog.golang.org/concurrency-is-not-parallelism where there is also a load balancer example around 20:20 and using go channels (the primitive I’m pertaining to)
> After playing with professional Load Balancers like NGINX I tried creating a simple Load Balancer for fun.
And while nginx[0] certainly can perform in this role, another production quality load balancer is HAProxy[1]. Both can do more than this, of course.
Reinventing solutions "for fun" certainly can be educational and help others learn key concepts, but the author should clearly state what they are doing is not meant to replace production quality solutions.
If you know you need a load balancer you should already know this.
> If you know you need a load balancer you should already know this.
If you know you need a load balancer, you go and look for one. If you happen upon one in a programming language you use, then it may be more appealing.
Hence the need for an explicit disclaimer.
Depends on the person and their understanding of English colloquialisms I suppose.
What might be less misunderstood is:
This is an educational project and is not
meant for use in production systems.
But why leave it to chance?
Amongst the synonyms for "dumb"[0] is "simple" and is used commonly as such.
A definition of "simple" is[1]:
easy to understand, deal with, use, etc.:
a simple matter; simple tools.
The first non-title line of the GitHub repo[2] reads thusly:
Simple LB is the simplest Load Balancer
ever created.
All of this is to say, what "might count for something" may not be what the author intends.
0 - https://www.merriam-webster.com/thesaurus/dumb
And here's the full list of synonyms listed there:
> airheaded, birdbrained, bonehead, boneheaded, brain-dead, brainless, bubbleheaded, chuckleheaded, dense, dim, dim-witted, doltish, dopey (also dopy), dorky [slang], dull, dunderheaded, empty-headed, fatuous, gormless [chiefly British], half-witted, knuckleheaded, lamebrain (or lamebrained), lunkheaded, mindless, oafish, obtuse, opaque, pinheaded, senseless, simple, slow, slow-witted, soft, softheaded, stupid, thick, thick-witted, thickheaded, unintelligent, unsmart, vacuous, weak-minded, witless
I am not racing through chains of synonyms.
What I was trying to elucidate was that the use of "dumbest", when the repo is ".../simplelb", and the very first line of the README says the project "is the simplest Load Balancer ever created" might, just might, lead people to think that "dumbest" is being used in this context as a synonym for "simplest".
Which might, just might, cause people to consider it for uses this project is not intended to satisfy.
Very strange axe you have to grind for a very unlikely hypothetical. Just because something is possible doesn’t mean it is probable.
Besides, even if a beginner somehow wanders into the project, mis-IDs the project, and uses on their beginner server, then it sounds like a good learning experience for them even though the README didn't give them permission to learn by neglecting to include the word "educational", god forbid.
Is this the catastrophic scenario you have in mind?
I was trying to be explicit as to my reasoning. If that came across as "hair-splitting" then I suppose I failed to adequately do so.
The whole point I was trying to make is that people find code in all sorts of ways. And my opinion is that if a public repo, such as GitHub, has a project which could easily be both desired (due to need) and misused (due to intent), then it might be a good idea to put a simple declaration in the project's README.
Given all the blow-back this concept has incurred, I would think the concept is either wholly immaterial or now proven as needed.
This seems to be a strange hill to die on. For all intents and purposes there is a declaration in the readme. And anyone who knows enough to want to operate with this will see that it's fairly basic. Others will likely just reach for a more generic or battle tested solution.
In any case, people should be able to do what they want with their repos and code, assuming legality of course.