Hacker News Reader

Comment by morning-coffee | Hacker News Reader

Didn't really convey where the magic happens to get whatever frames are produced by the userspace TCP/IP stack to whatever mechanism puts those frames on some wire... isn't it still making some syscall? IOW, where's the I/O happening?

View on HN

I haven’t followed for a long time but last time I checked the state of the art was getting your packet in and out via an eBPF module like XDP actually bypassing the kernel network stack (that’s not what the article is doing). You would then be directly talking to the NIC through the module which is akin to sending packets directly through the wire.

Theres also the people that just map the PCIe device into a userland process: https://www.dpdk.org/

You can't really just map a PCIe device into userland (by that I mean that the just here does a lot of work). As far as I know, DPDK uses either a BSD NIC driver running in userland (somewhat slow) or XDP on Linux.

No, that is indeed the working principle (plus minus some yak shaving with iommus and what not). It certainly doesn't use XDP, that would defeat the purpose.

I don’t see in anyway how it would defeat the purpose especially considering it certainly does use XDP [1] for everything it doesn’t have a driver for and you can opt for it instead of an existing driver if you actually want to have something more performant.

[1] https://doc.dpdk.org/guides/nics/af_xdp.html

Also, "no" about what? DPDK doesn’t just expose the PCIE device to user land. There is what you call the yak shaving with memory and then it also has a full driver from the NIC so you can actually you know do networking.

[1] https://doc.dpdk.org/guides/nics/af_xdp.html

The simplest way to do this, and the way I think this post is referring to, is to bring up a WireGuard session. WireGuard is essentially just a UDP protocol (so driveable from userland SOCK_DGRAM) that encapsulates whole IP packets, so you can run a TCP stack in userland over it.

Thanks. I read a bit about WireGuard... seems to create a tunnel over UDP as you say. Where I got off track is the following comment in the article:

> Who says you need to do networking in the kernel though?

I thought it implied they were bypassing the kernel entirely. This isn't true at all if it's still using a UDP socket to send... there's still a ton of code in the kernel to handle sending and receiving UDP. It's also a helluva lot less efficient in terms of CPU consumed, unless one is using sendmmsg, because it's a syscall per tiny datagram between user space and kernel space to get the data onto the UDP socket...

When people talk about bypassing the kernel, they mean taking control of TCP/IP (UDP, IPv6, ICMP, whatever) in userland; for instance, doing your own congestion control algorithm, or binding to every port on an address all at once, or doing your own routing --- all things the kernel doesn't expose interfaces for. They usually don't mean you're literally not using the kernel for anything.

There are near-total bypass solutions, in which the bulk of the code is the same as you'd use with this WireGuard bypass, where ultra-high performance is the goal, and you're wiring userland up to get packets directly to and from the card DMA rings. That's a possible thing to do too, but you basically need to build the whole system around what you're doing. The WireGuard thing you can do as an unprivileged user --- that is, you can ship an application to your users that does it, without them having to care.

Long story short: sometimes this is about perf, here though not so much.

To me the key thing with any TCP implementation is the timers. Just sending a single TCP packet causes the kernel to have to track a lot of state and then operate asynchronously from your program to manage this state, which also requires the kernel to manage packet buffering and queuing for you.

If you're interested in the latter technique (hitting the HW directly from userspace), take a look at https://www.dpdk.org/. That's the framework that 95% of things that do that use (number pulled directly out of my rear end).

> If you're interested in the latter technique (hitting the HW directly from userspace), take a look at dpdk.org

I hope, by "you", you mean the rest of us journeymen and not this tptacek [0]

[0] https://fly.io/blog/bpf-xdp-packet-filters-and-udp/ / https://archive.md/TVIyy

None of us can ever know ahead of time what techs a stranger has seen. So, it’s fine to say “you might find this helpful.” They will or they won’t but it’s still a kind act.

I was hoping someone else was going to call out DPDK, because I've never used it, am aware of it, and don't want to explain it. :) You can't possibly offend me by suggesting I'm not aware of some technology.

I've never used dpdk for kernel bypass, only open onload. Originally for solarflare hardware, apparently it works with cards from other vendors via XDP (though I've never tried it). It gets loaded via LD_PRELOAD and replaces ordinary BSD socket functions with full userspace sockets that can fall back to the kernel when necessary (e.g. if the application wants to yield the cpu but still get notified about the socket).

https://github.com/Xilinx-CNS/onload

Also: packet parser vulnerabilities.

Traditionally on Linux it's going to be AF_PACKET sockets: you can read and write raw ethernet frames via the usual syscalls.

https://www.man7.org/linux/man-pages/man7/packet.7.html

There are more modern interfaces which perform better.

Writing raw ethernet frames, can you do that as an unprivileged user? Or is the point more that we're doing userspace networking, which could nevertheless be run as root?

You need CAP_NET_RAW, but I'm pretty sure you can drop privileges after bind()

Reply on news.ycombinator.com