Hacker News Reader

Comment by tptacek | Hacker News Reader

The simplest way to do this, and the way I think this post is referring to, is to bring up a WireGuard session. WireGuard is essentially just a UDP protocol (so driveable from userland SOCK_DGRAM) that encapsulates whole IP packets, so you can run a TCP stack in userland over it.

View on HN

Thanks. I read a bit about WireGuard... seems to create a tunnel over UDP as you say. Where I got off track is the following comment in the article:

> Who says you need to do networking in the kernel though?

I thought it implied they were bypassing the kernel entirely. This isn't true at all if it's still using a UDP socket to send... there's still a ton of code in the kernel to handle sending and receiving UDP. It's also a helluva lot less efficient in terms of CPU consumed, unless one is using sendmmsg, because it's a syscall per tiny datagram between user space and kernel space to get the data onto the UDP socket...

When people talk about bypassing the kernel, they mean taking control of TCP/IP (UDP, IPv6, ICMP, whatever) in userland; for instance, doing your own congestion control algorithm, or binding to every port on an address all at once, or doing your own routing --- all things the kernel doesn't expose interfaces for. They usually don't mean you're literally not using the kernel for anything.

There are near-total bypass solutions, in which the bulk of the code is the same as you'd use with this WireGuard bypass, where ultra-high performance is the goal, and you're wiring userland up to get packets directly to and from the card DMA rings. That's a possible thing to do too, but you basically need to build the whole system around what you're doing. The WireGuard thing you can do as an unprivileged user --- that is, you can ship an application to your users that does it, without them having to care.

Long story short: sometimes this is about perf, here though not so much.

To me the key thing with any TCP implementation is the timers. Just sending a single TCP packet causes the kernel to have to track a lot of state and then operate asynchronously from your program to manage this state, which also requires the kernel to manage packet buffering and queuing for you.

If you're interested in the latter technique (hitting the HW directly from userspace), take a look at https://www.dpdk.org/. That's the framework that 95% of things that do that use (number pulled directly out of my rear end).

> If you're interested in the latter technique (hitting the HW directly from userspace), take a look at dpdk.org

I hope, by "you", you mean the rest of us journeymen and not this tptacek [0]

[0] https://fly.io/blog/bpf-xdp-packet-filters-and-udp/ / https://archive.md/TVIyy

None of us can ever know ahead of time what techs a stranger has seen. So, it’s fine to say “you might find this helpful.” They will or they won’t but it’s still a kind act.

I was hoping someone else was going to call out DPDK, because I've never used it, am aware of it, and don't want to explain it. :) You can't possibly offend me by suggesting I'm not aware of some technology.

I've never used dpdk for kernel bypass, only open onload. Originally for solarflare hardware, apparently it works with cards from other vendors via XDP (though I've never tried it). It gets loaded via LD_PRELOAD and replaces ordinary BSD socket functions with full userspace sockets that can fall back to the kernel when necessary (e.g. if the application wants to yield the cpu but still get notified about the socket).

https://github.com/Xilinx-CNS/onload

Also: packet parser vulnerabilities.

Reply on news.ycombinator.com