Snap: A microkernel approach to host networking (2019) | Hacker News Reader

Snap: A microkernel approach to host networking (2019)

research.google

85 pointsteleforce13 comments

So much interesting stuff happening in userspace networking these days. It really feels like the pieces are almost there to treat the network as just another part of the stack that you're programming. (Albeit a pretty low-level part!)

If you're interested in Google Snap, you might also want to take a look at Snabb [0], and Rush (Snabb in Rust) [1].

For an example of how this stuff is useful at sub-Google scale, we just open-sourced the Rush-based tools we use to test our WebRTC code [2]

[0] https://blogs.igalia.com/dpino/2017/11/13/snabb-network-tool...

[1] https://github.com/eugeneia/rush

[2] https://github.com/daily-co/synthetic-network

Rush sounds interesting, but the README is pretty sparse. Do you know more context/background?

Sure. We [0] make APIs for live video and audio, so we do a lot of different kinds of network testing. Over time, we had cobbled together a variety of tools to support load/scaling testing, CI/CD, testing during development, and helping our customers debug and test their applications. For all of these, some ability to simulate packet loss, latency, jitter, and other network realities is somewhere between helpful and critical.

I'm a big fan of Snabb, and had a few conversations with Max Rottenkolber, one of the Snabb authors, about how we might tackle building a common network-simulation framework that we could use across all the different kinds of testing we do. Max suggested more or less the approach that you can see in the synthetic-network repo. (He wrote a series of documents during implementation that are really fun reading. They are in the /doc directory.)

Since we increasingly use Rust in various places, and our use case was a bit different than the core use cases for Snabb, Max took a swing at porting parts of Snabb to Rust for the synthetic-network project. That worked out really well, and that Snabb-implemented-in-Rust is now Rush!

[0] https://www.daily.co/

> So much interesting stuff happening in userspace networking these days. It really feels like the pieces are almost there to treat the network as just another part of the stack that you're programming. (Albeit a pretty low-level part!)

There’s also fly doing userland tcp over userland wireguard because peeps might not have a local wireguard client.

It’s not just Fly IIRC, the official macOS Wireguard also relies on userland Wireguard (wireguard-go), as does Tailscale.

In addition to userland Wireguard, we (Tailscale) also increasingly use Google's userland TCP/IP stack (netstack) for platforms that lack kernel facilities we need, when drivers aren't available, or when we lack permissions (e.g. user not running as root, or running as some weirdo root in some container environment).

> There’s also fly doing userland tcp over userland wireguard because peeps might not have a local wireguard client.

Oh, yeah! That whole thing [0], up through the integration with flyctl, is fire.

[0] https://news.ycombinator.com/item?id=26315695

I was reading a blog post ~6 months ago, details are sparse in my head but it was some sort of cloud compute company that used userspace wireguard for users to be able to ssh into the virtual infrastructure. Keys were managed so users didn't have to manage a system wireguard.

This reminded me of my bitbucket CI/CD which didn't have any way to connect into our internal infrastructure to drop of built artifacts. Bitbucket runs their docker containers in a way that blocks you from being able to run wireguard or zerotier.

One weekend I played around with the Python bindings to ZeroTier, and their "public" networks, to create a proxy client that SSH could use, given an SSH private key and ZeroTier network ID, to (in userspace) SSH to a remote server behind our firewall via ZeroTier.

I haven't used it in production yet, but it seemed to work in my tests. Now they have Bitbucket build clients you can run on your own infrastructure, so maybe that would allow us to do those builds from inside our network, so we could push things to our deploy servers.

It was us! https://fly.io/blog/ssh-and-user-mode-ip-wireguard/

We do more now. Our CLI uses the stack to talk to remote Docker daemons for builds, nats for log streaming, a private DNS service, etc. It's very handy.

Ooh, thanks for the pointer! Fellow HNers: Check out that blog post. It kind of blew my mind, very clever use of user-mode VPN! Got my mind racing a bit coming up with ways I could do something similar.

Earlier discussion (2019 Oct / 144 points / 75 comments): https://news.ycombinator.com/item?id=21374204

How does this compare to OpenOnload?

Hard comparison to make. Snap can be used with traditional socket APIs, but that is not really its purpose. OpenOnload can only be used that way.

Reply on news.ycombinator.com