Linkerd-tcp: A lightweight, service-discovery-aware, TLS-ing TCP load balancer

So far latency has been comparable with other native load balancers: we've built performance profiles for nginx proxy_pass and haproxy to compare against although as we add features our numbers are starting to come down a bit.

We're still building our collection of tools for tracking down performance issues like lock contention, allocation hotspots, etc. I've had good luck finding cpu hotspots with linux perf and valgrind but less luck tracking down mutex holders and allocation hotspots. Any advice is welcome!

We do not preserve source IP currently.

linkerd is a router and not a "web server", so there may be features web-servery features that you still want from nginx that don't belong in linkerd. However, linkerd should be useful as an ingress server. To this end, we've just recently added support for the kubernetes Ingress API, which is available in 1.0-RC1.

It's tough to compare the performance of the two systems because linkerd-tcp is not an HTTP proxy currently, it's a TCP proxy. That said, the resource footprint has been much lower than a finagle service. My first prototype, which had no fancy features like load balancing or metrics, was able to proxy a gigabit worth of traffic in less than a core on an older Ivy Bridge machine and used about 10Mb of RSS.

The biggest challenge has been learning to use the ownership system properly. It took me a few frustrating weeks to learn a new set of patterns and techniques for structuring code in a Rust-friendly method. Reading through the tokio code and example projects was really helpful in that regard.

My team at Apprenda built the first K8s distro/toolkit (0) with support for Linkerd last year.

0: https://github.com/apprenda/kismatic/blob/master/docs/LINKER...

We sometimes proxy to another DC across the Internet and haven't vetted HAProxy's SSL support thoroughly. So we're running Nginx to terminate SSL, then to HAProxy, then to an stunnel instance per backend server. To get keepalives working and increase the throughput of the solution we've worked a bit on the Nginx config and put HAProxy into tunnel mode. We don't rewrite any headers or anything of that nature at the HAProxy link of this chain, so we don't need anything more. With proxying through all that, our page load times are basically indistinguishable from hitting a server directly unless it goes to the far data center. All the coordination is handled by a Puppet module so the complexity isn't that scary.

If Linkerd-tcp can top that or even come close with all the automation and integration in its ecosystem, though, it's a definite reason to take a hard look at it. Getting the Prometheus and Graphana output really simply is a great benefit.