Hacker News Reader

Comment by londons_explore | Hacker News Reader

At cloudflare-scale, I think I would want a proxy that sits at least partially in the network hardware.

Ie. so that the bytes of a large image or video that is coming from an origin server and being sent to a client never has to pass through the system RAM or CPU.

I'd probably implement this as mods to the sendfile() API so that a certain number of bytes can be copied from one socket to another, with that request going all the way to the firmware of the network card which will do the actual work.

It probably needs to work with HTTP/3 UDP and encryption too - so decrypt the data from this socket and send it to that socket reencrypted with this other key and packetized to this HTTP/3 stream. The firmware would need some way to do aborts/timeouts and kick a partially complete request back to software too.

Is it complex...? Yes. But will the compute savings be worth it...? At cloudflare scale, I think the answer is yes.

View on HN

We constantly think about the possibilities of using exotic hardware for acceleration. So far we've got a very, very long way with "commodity" hardware and the Linux kernel. One day, we'll probably do something exotic.

https://blog.cloudflare.com/cloudflares-gen-x-servers-for-an...

https://blog.cloudflare.com/tubular-fixing-the-socket-api-wi...

Just curious, is there a high cost overhead (aside from the cost of actual units) of e.g. custom silicon?

It’s just a question of “is it worth it?”. How much does this cool accelerator thing cost? How much does it save?

Why not using the same kind of solution as Netflix does where they offload TLS and the like in hardware?

https://www.nvidia.com/en-us/networking/ethernet/connectx-6-...

Pretty sure they analyse TLS handshakes for their Bot Management product and probably other things.

Have you tried out io_uring? My impression is that these Rust libs end up making different syscalls that are less great.

Tokio supports io_uring (https://github.com/tokio-rs/tokio-uring), so perhaps when it's mature and battle-tested, it'd be easier to transition to it if Cloudflare aren't using it already.

Don't assume io_uring is an automatic win.

Last time I benchmarked io_uring, it was good enough, but I was able to get measurably faster from syscalls and a well-crafted userspace thread pool for storage to NVMe.

(Note: the I/O thread pool I wrote came out noticably faster than fio, the benchmark tool, so don't assume fio results are the best possible.)

For that reason, I have a small application which can be configured for either io_uring or the thread pool, and the thread pool is preferred for performance.

I don't think the hardware need be very exotic - most network hardware already has firmware capable of various types of acceleration, and all you would have to do is partner with the network card vendor and have a joint team write firmware that has all the features you need.

With shared source, the network card vendor can even repackage some of the types of acceleration you write for their other clients - and I bet they're already looking into acceleration of HTTP/3.

It's a win-win - because cloudflare gets massive compute savings, and the network card vendor knows they've built quite a high moat to buying someone else's hardware.

If someone from Cloudflare is reading... please don't do this. I'm asking for two reasons:

1. Cloudflare has been great at rapidly iterating (compare the timeline between HTTP/3 support in Cloudflare vs Cloudfront... let alone AWS's ALB that still doesn't support it). Introducing hardware accelerators would surely hinder those efforts (ouch, we need Y to do this that the hardware can't do, and the vendor says it'll take 1 year to have new production-ready cards).

2. Cloudflare has been a good upsteram contributor for the projects they depend on (the kernel, the rust language, etc.). Partnering with a hardware vendor inevitably means closed source, deviations from upstream and ultimately a much larger hurdle to be a good citizen to the open-source community.

As you can see, the reasons are entirely selfish. I'd understand it if you do it anyways because the numbers make sense for your company. In the meantime... thanks for holding out until you really can't justify it anymore!

Those are actually reasons I'm hesitant. We want to be able to support protocols etc. very quickly and can't wait for hardware to catch up. As I said elsewhere the combination of CPUs and the Linux kernel has worked pretty well for us.

> Partnering with a hardware vendor inevitably means closed source, deviations from upstream and ultimately a much larger hurdle to be a good citizen to the open-source community.

No, it absolutely doesn't "automatically" mean that. You can totally request some major hardware vendors to fully upstream hardware offload capabilities in their Linux drivers.

Cloudflare has generally been very conservative about its tech stack (eg kernel TCP, well-known OSS solutions and abstractions), which seems to work fine for them. They are certainly leaving performance on the table, but they also don't have to pay to support exotic solutions.

Other firms in the same space do things that are a lot more exotic, and it is worth it for them.

Reply on news.ycombinator.com