Comment by johnousterhout | Hacker News Reader

Comment by johnousterhout | Hacker News Reader

Parent Full thread

Just to set the record straight:

1. Homa detects and recovers from whole-RPC loss just fine; always has.

2. Homa does not have built-in encryption support (any more than TCP does), but encryption can be (and has been) layered on top pretty easily with DTLS. Isn't that what QUIC does also (layer encryption on top of UDP)?

3. Performance: I think you may be misreading the figure. For W4 or W5, Homa uses about 5 cores to get a total of 40 Gbps (20 Gbps in each direction), which is about 8 Gbps/core. I think you'll be hard-pressed to find a software transport that does much better than this (make sure you are measuring mixed workloads with a variety of message sizes; if you only send large messages then you can do much better (and so does Homa)). Unfortunately, all software transport protocols are resource hogs; in my opinion, transport implementation really should move to the NIC.

4. I would be interested to see measurements of QUIC performance running the same workloads as Homa. My understanding is that QUIC has all of the congestion control problems of TCP, so I'd expect it to perform like TCP under mixed workloads (i.e. 10-100x longer tail latency than Homa for short messages). Among other things, I don't believe QUIC uses switch priority queues, so it does not "solve basically every problem Homa is trying to solve".

1. Upon careful re-reading I see:

“Ultimate responsibility for RESENDs lies with the client: if it has not received the entire response message within an expected time then it MUST issue RESEND request(s) for the missing part(s) of the response.”

I retract my statement that it does not describe whole RPC loss in protocol.

However, even as someone familiar with network protocols and reading RFCs that is a very oblique and easy to miss way to describe the process. I would recommend a more direct and explicit treatment of the “request lost” case and make clear the “RESEND Length=-1” mechanism as this seems like the only case where that should be relevant. Likely also a state machine since the state flows are different than the regular data loss mechanism; Tx DATA, Tx Lost, Tx RESEND, Rx RPC_UNKNOWN, Tx DATA or something of the like.

Also as I mentioned in another reply [1] the described mechanism has many undesirable propertys.

2. My statement was more about explicit consideration as to how encryption integrates. Sure, you can just layer it below with IPSec or above with TLS, or any other myriad of ways as you could in literally any protocol for transmitting bytes, but TLS over TCP versus QUIC shows how much more painful it is if it was not in the original design.

3. I did not see that it was a duplex benchmark, but that is a very non-standard way to benchmark, so I can not comment as to comparability to standard one-way benchmarks. If a one-way benchmark is equivalent to the sum of the two-way flows, then I will concede that it is comparable to QUIC implementations.

However, my personal experience with designing and implementing transport protocols results in me finding 8 Gbit/s per core in software to be extremely slow which also applies to every QUIC implementation I have ever seen.

4. Congestion control in QUIC, like in TCP, is not fixed though there are recommended implementations. Furthermore, stream prioritization, something not present in TCP due to the single stream nature, is also not fixed. A implementation that offers prioritizing which streams to transmit data for by SRPT would likely see similar outcomes to Homa. I am not aware of any implementation that does so, but it is a freely configurable element of that protocol. I see nothing that would materially stop a comparable solution from being dropped in.

Mainly, unlike TCP, the design of QUIC has knobs in many of the right places so that you can reasonably configure the desired propertys with minimal design clashing.

Really, the main fundamental distinction in steady state operation is the resend mechanism being largely ack driven TCP and QUIC whereas Homa is largely nack driven.

I am of the opinion that ack driven is much superior technically, which is part of why I think Homa is a bad design, but likely for different reasons and considerations as to how it relates to transport protocol design than the designers of QUIC.

This is also not to say that I think QUIC is the best protocol, QUIC has a number of design and performance weaknesses, but it is overall a reasonable and material improvement over TCP and should be considered the standard to beat.

[1] https://news.ycombinator.com/item?id=49960483

My knowledge of QUIC is pretty superficial, but it sounds like you are saying that congestion control is not a fundamental property of QUIC, so someone could potentially implement Homa's congestion control mechanism in QUIC. If someone were to do that, then of course I'd expect that to perform similarly to Homa.

However, this would create a conflict within QUIC. My understanding is that one of QUIC's primary use cases is WAN connections between browsers and Web servers. In this world Homa's congestion control mechanism does not work; it only works in datacenter environments with speed-of-light delays at most a few 10's of microseconds. Thus, QUIC would need to support multiple congestion control mechanisms, one for WAN and one for datacenter and choose between them as appropriate.

In addition, my understanding is that QUIC is stream-based, which would mean, as with TCP, the protocol doesn't know about message lengths. This makes it hard to prioritize short messages.

And, by the way, as much as you may prefer acks to grants, the grant approach is pretty fundamental to Homa's congestion control.

To me, QUIC and Homa were designed for very different environments. Homa will never compete with QUIC in the WAN, and I think it will be hard for QUIC to compete with Homa in the datacenter (Homa can specialize itself entirely for datacenter apps, whereas QUIC can't).

You should do a deeper dive into QUIC, it has a number of interesting ideas.

1. Having multiple congestion control algorithms is already standard in both QUIC and TCP. As deploying Homa requires you to control both ends or at least understand both ends, in a comparable QUIC deployment you would know they are in a datacenter and can just link the corresponding implementation. Or you could use the configuration parameter handshake mechanism to switch at runtime for certain flows.

2. QUIC is stream-based, but they support fixed/final-sized streams which are logically equivalent to messages. You just generate a disposable fixed-size stream for each “message”. So, it is trivial for the Tx side to know the “message” size and thus do SRPT prioritization. The thing that would be more annoying to support is the Rx side knowing the stream size as size finalization is detected by the Rx side via the FIN bit when transmitted the terminal segment in a stream. You could transmit the terminal segment first, but that is awkward and Homa is superior in that respect. However, Rx size finalization would only really be important for avoiding over-allocation of Rx buffering in QUIC since they do not rely on MAX_STREAM_DATA for flow control whereas the equivalent concept in Homa, GRANT, is used for flow control and thus precise Tx sizing awareness by the Rx side is more important.

3. As mentioned above, Homa GRANT is very similar to QUIC MAX_STREAM_DATA which are basically just extensions of TCP window size to more than one data flow. The main difference is how they are used.

TCP and Homa both use it for Rx resource availability and bandwidth limiting. Homa uses it additionally for data flow prioritization as it has more than one data flow. QUIC uses it for Rx resource availability and implementations can use it for data flow prioritization. QUIC uses MAX_DATA or just plain congestion control bandwidth estimation for bandwidth limiting.

The fundamental usage is Rx resource availability. If you have no Rx buffers available to take the data, then no amount of data sent is meaningful. That is a hard cap. However, using that as a bandwidth limiting mechanism like TCP and Homa requires continuous tight feedback to avoid the Tx from determining there are no Rx resources available while not over-provisioning allowed bandwidth.

To illustrate, suppose I have a 10 GB/s NIC and generate a 1 GB request per second, but my end-to-end is a 1 GB/s link with 1 ms one-way latency. So, this link only handles 1 MB in-flight without buffering. For the sake of simplicity say we pack only 1 KB per packet, so the link can only have 1,000 packets in flight without buffering. How should we manage the GRANT flow?

If we do a initial GRANT of 10 MB, then the Tx will shovel 10 MB into its 10 GB/s NIC which will spit out into the network in 1 ms. This will clog up at the 1 GB/s bottleneck requiring ~10 MB of buffering at the bottleneck. The Tx will then wait for more GRANT as it just bursted it all. After 1 ms the Rx will get the first byte, what should it GRANT and when? If we wait 1 ms, we have received our first MB, should we GRANT 1 MB? Our steady state then looks like bursts of 10 GB/s with 10 MB of buffering at the bottleneck until the request completes.

If we do a initial GRANT of 1 MB then GRANT 1 MB when we get it all, then we stall.

If we do a initial GRANT of 2 MB, then 1 MB every ms then we get 2 MB of buffering at the bottleneck until the request completes.

What is the optimal flow? The optimal outcome is a continuous 1 GB/s flow with one 1 KB packet entering every 1 us. This results in minimal buffering as a result of the transport protocol. For the GRANT mechanism to do this would require a new 1 KB GRANT to arrive every 1 us.

Absent a separate point-to-point bandwidth control system to pace the DATA flow you are not getting that. But once you have that, the GRANT is no longer the bandwidth limiter, it becomes a Rx resource availability and flow prioritization mechanism, just like QUIC. It no longer becomes meaningfully distinct.

In your item 2, it's important for the receiver to know the message size not just for allocating Rx resources, but also for prioritizing its incoming messages.

The problem you discuss in item 3 (an intermediate link much slower than the host uplinks) seems like something that will only occur in a WAN. In datacenters the host uplinks are the slowest links, so I don't think this is an issue for Homa.

Item 3 is when bandwidth available to a specific flow in a intermediate uplink is limited. This occurs even if you have a 1 TB/s intermediate link as long as there are 1000 1 GB/s flows (of potentially heterogeneous origin) that go through that link.

Unless every intermediate link has more bandwidth than the sum of every flow that goes through it, you have this problem. Thinking about that more, that is literally just “congestion is possible”. But if congestion is not possible, then there is no bandwidth limiting and you can just blast away at full speed at all times.

I guess I would characterize this as congestion control/bandwidth limiting where your control parameter is “in-flight data allowed”, which is what TCP also does, results in exactly that. You get intermediate buffering due to no pacing mechanism unless you are giving rapid, continuous, precise congestion control feedback. The control parameter that avoids that is available bandwidth (or estimated available if you are not omniscient) which allows the Tx to interpolate between explicit congestion control feedback. This allows reduction in rapidity of congestion control feedback while neither stalling nor over-producing. You just tradeoff responsiveness to channel parameter changes in proportion to how rapidly you provide explicit congestion control feedback.

Data flow prioritization within your available bandwidth is, however, still necessary and largely orthogonal to the above.

Reply on news.ycombinator.com