5,121 karma · joined May 4, 2019
It is a mechanism that is far more practical and likely to succeed at broad reform than their plan of convincing magazine editors and instagram models.
All it would need to do is not highlight “improved spellings” as incorrect in “English 20xx” mode at the start. Then it could slowly turn the knob as they see increasing usage.
If you want to do even more then you can implement speech-to-text and transcript generators that default to “English 20xx”.
The output program does what the input source code does. The abstraction is a precise translation and non-leaky, it just admits multiple possible internals. That does not make the translation any less precise.
All observable and defined side effects should be identical otherwise you have a compiler bug. Optimizations that are not generically semantics preserving may be used, but should only be used where the program is semantically identical for your use case otherwise you have incorrectly used the options available in your compiler.
Thankfully, legal systems are not that inhumanly stupid.
I am pointing out how that is a absurd moral argument deriving from the rule of thumb. If anything, the corollary is the literal opposite.
Any stupidity indistinguishable from malice should be treated the same.
The usage as a epistemological rule of thumb just points out how great harm is usually caused by stupidity, intentional or otherwise, rather than intentional desire to cause the specific harm. If you aim to reduce great harms targeting just malice is not merely inadequate, it is likely missing the overwhelming majority.
If harms are excused because they were not made with evil intentions then the Nazis who just liked making the numbers go up in a orderly fashion should be considered blameless.
> Every Tesla ever made has a door release that doesn't require any electricity.
Was a valid response to the comment you replied to:
> Like the people who get trapped in case of accident when the electricity fail and is burned inside?
Implying others are making untrue statements by making intentionally deceptive statements is very rude.
As for the ad provider, it is trivial to not deliver criminal payloads if your business starts from the assumption that doing so is costly instead of something you can foist on society for your own benefit.
Why would they feel the need to handle garden variety illegal behavior like school speed limits?
The only advantage is that cloud services provide VM tenants so you are already stuck with a VM substrate and you want to minimize layers from that point on.
However, as a practical matter, the design of the Linux kernel is grossly insecure and basically impossible to make even remotely safe as evidenced by the unending sprawl of LPEs. Hypervisors are generally designed and configured in a way that is only mostly insecure instead of grossly insecure; a bad is better than terrible situation.
However, there is nothing stopping you from designing a kernel in a similar way and providing shared tenant processes with similar guarantees. You just need to port your applications to the new operating system, but you would also need to port them to a unikernel anyways if you wanted to go that route.
The only free action is lumping your applications with the full OS and dumping it onto a hypervisor. Anything else has porting work which is why the multi-tenant VM model is advantageous at all in the first place.
I was explaining why undefined behavior as a concept is a very sensible idea. Whether the expansive interpretation of the optimizations you are allowed to do when encountering the “impossible” are reasonable is a different question.
if (x > 0) {…}. But what if you entered the body when x <= 0?
if (false) {…}. But what if you execute the body?
These are “impossible”. What happens when the impossible occurs is “undefined”.
When the older standards said signed integer overflow for addition is undefined what they are actually saying is that the real definition of + is:
int +(int x, int y) {
assert(in_range(actual_math_add(x, y), signed_int_min, signed_int_max));
return machine_add(x, y);
}
But, in the interest of performance, “release mode” (which in this case is just any compilation) elides asserts since as a programmer you should not write code that asserts in much the same way that you should not write assert(false) in a normal code path that is supposed to run. Assertions are intended for “impossible” code paths and usually get compiled out in “release mode” though maybe your code is buggy and can actually hit them and then your program goes off the rails because it had a bug.
Put another way, if you did write assert(false) in a regular code path, would you find it unreasonable for the compiler to just delete the code after it? That is what undefined behavior is for.
Unless every intermediate link has more bandwidth than the sum of every flow that goes through it, you have this problem. Thinking about that more, that is literally just “congestion is possible”. But if congestion is not possible, then there is no bandwidth limiting and you can just blast away at full speed at all times.
I guess I would characterize this as congestion control/bandwidth limiting where your control parameter is “in-flight data allowed”, which is what TCP also does, results in exactly that. You get intermediate buffering due to no pacing mechanism unless you are giving rapid, continuous, precise congestion control feedback. The control parameter that avoids that is available bandwidth (or estimated available if you are not omniscient) which allows the Tx to interpolate between explicit congestion control feedback. This allows reduction in rapidity of congestion control feedback while neither stalling nor over-producing. You just tradeoff responsiveness to channel parameter changes in proportion to how rapidly you provide explicit congestion control feedback.
Data flow prioritization within your available bandwidth is, however, still necessary and largely orthogonal to the above.
1. Having multiple congestion control algorithms is already standard in both QUIC and TCP. As deploying Homa requires you to control both ends or at least understand both ends, in a comparable QUIC deployment you would know they are in a datacenter and can just link the corresponding implementation. Or you could use the configuration parameter handshake mechanism to switch at runtime for certain flows.
2. QUIC is stream-based, but they support fixed/final-sized streams which are logically equivalent to messages. You just generate a disposable fixed-size stream for each “message”. So, it is trivial for the Tx side to know the “message” size and thus do SRPT prioritization. The thing that would be more annoying to support is the Rx side knowing the stream size as size finalization is detected by the Rx side via the FIN bit when transmitted the terminal segment in a stream. You could transmit the terminal segment first, but that is awkward and Homa is superior in that respect. However, Rx size finalization would only really be important for avoiding over-allocation of Rx buffering in QUIC since they do not rely on MAX_STREAM_DATA for flow control whereas the equivalent concept in Homa, GRANT, is used for flow control and thus precise Tx sizing awareness by the Rx side is more important.
3. As mentioned above, Homa GRANT is very similar to QUIC MAX_STREAM_DATA which are basically just extensions of TCP window size to more than one data flow. The main difference is how they are used.
TCP and Homa both use it for Rx resource availability and bandwidth limiting. Homa uses it additionally for data flow prioritization as it has more than one data flow. QUIC uses it for Rx resource availability and implementations can use it for data flow prioritization. QUIC uses MAX_DATA or just plain congestion control bandwidth estimation for bandwidth limiting.
The fundamental usage is Rx resource availability. If you have no Rx buffers available to take the data, then no amount of data sent is meaningful. That is a hard cap. However, using that as a bandwidth limiting mechanism like TCP and Homa requires continuous tight feedback to avoid the Tx from determining there are no Rx resources available while not over-provisioning allowed bandwidth.
To illustrate, suppose I have a 10 GB/s NIC and generate a 1 GB request per second, but my end-to-end is a 1 GB/s link with 1 ms one-way latency. So, this link only handles 1 MB in-flight without buffering. For the sake of simplicity say we pack only 1 KB per packet, so the link can only have 1,000 packets in flight without buffering. How should we manage the GRANT flow?
If we do a initial GRANT of 10 MB, then the Tx will shovel 10 MB into its 10 GB/s NIC which will spit out into the network in 1 ms. This will clog up at the 1 GB/s bottleneck requiring ~10 MB of buffering at the bottleneck. The Tx will then wait for more GRANT as it just bursted it all. After 1 ms the Rx will get the first byte, what should it GRANT and when? If we wait 1 ms, we have received our first MB, should we GRANT 1 MB? Our steady state then looks like bursts of 10 GB/s with 10 MB of buffering at the bottleneck until the request completes.
If we do a initial GRANT of 1 MB then GRANT 1 MB when we get it all, then we stall.
If we do a initial GRANT of 2 MB, then 1 MB every ms then we get 2 MB of buffering at the bottleneck until the request completes.
What is the optimal flow? The optimal outcome is a continuous 1 GB/s flow with one 1 KB packet entering every 1 us. This results in minimal buffering as a result of the transport protocol. For the GRANT mechanism to do this would require a new 1 KB GRANT to arrive every 1 us.
Absent a separate point-to-point bandwidth control system to pace the DATA flow you are not getting that. But once you have that, the GRANT is no longer the bandwidth limiter, it becomes a Rx resource availability and flow prioritization mechanism, just like QUIC. It no longer becomes meaningfully distinct.
In a lossless RPC the client should expect the reply in 1002 ms. When should the client resend the request if it got lost?
Given that the client would not expect to see the first byte of the reply until 1002 ms and receives no other feedback until then, they should not issue a resend of their request until 1002 ms.
That means if the network lost their first request, they will not send a resend until 1002 ms and then get the reply to their resend at 2004 ms.
In contrast, in a ack based approach, the client would expect a server ack in ~2 ms as the server gives feedback on receipt rather than on processing completion/reply start.
That means if the network lost their first request they will not send a resend until 2 ms and then get the reply to their resend at 1004 ms, nearly half the latency.
In ack based approaches, packet loss requires one additional RTT latency to recover per repeated packet loss. But Homa incurs one additional RTT + processing latency to recover per repeated packet loss.
As a secondary downside, this requires the client to hold Tx resources for RTT + processing time since the client does not know when the server has received all of the data. In ack-based approaches, you learn the server has received all of the request when it acks all of the data in the request which is on the order of the RTT and thus can release Tx resources at that time. This is similar to the reason Homa has ACK/NEED_ACK packets except applied to client side resources instead of just server side resources.
The case you are describing is more like what if the server explodes, but you still need the results/modifications related to your RPC to occur. In that case, whether the Tx got through or not is irrelevant because your RPC still did not complete and you need it to complete, so there is no advantage to the ack-based approach. But by that logic, what if it explodes before persisting changes? Or what if all your servers explode? At some point we should call that out of scope for a transport protocol.
I prefer setting that line at “bytes got from one end to the other”. If you want higher guarantees then you can layer that. As you point out, it is easy to just layer a “server dead” timeout ping which basically solves the problem in the same way Homa needs to.
“Ultimate responsibility for RESENDs lies with the client: if it has not received the entire response message within an expected time then it MUST issue RESEND request(s) for the missing part(s) of the response.”
I retract my statement that it does not describe whole RPC loss in protocol.
However, even as someone familiar with network protocols and reading RFCs that is a very oblique and easy to miss way to describe the process. I would recommend a more direct and explicit treatment of the “request lost” case and make clear the “RESEND Length=-1” mechanism as this seems like the only case where that should be relevant. Likely also a state machine since the state flows are different than the regular data loss mechanism; Tx DATA, Tx Lost, Tx RESEND, Rx RPC_UNKNOWN, Tx DATA or something of the like.
Also as I mentioned in another reply [1] the described mechanism has many undesirable propertys.
2. My statement was more about explicit consideration as to how encryption integrates. Sure, you can just layer it below with IPSec or above with TLS, or any other myriad of ways as you could in literally any protocol for transmitting bytes, but TLS over TCP versus QUIC shows how much more painful it is if it was not in the original design.
3. I did not see that it was a duplex benchmark, but that is a very non-standard way to benchmark, so I can not comment as to comparability to standard one-way benchmarks. If a one-way benchmark is equivalent to the sum of the two-way flows, then I will concede that it is comparable to QUIC implementations.
However, my personal experience with designing and implementing transport protocols results in me finding 8 Gbit/s per core in software to be extremely slow which also applies to every QUIC implementation I have ever seen.
4. Congestion control in QUIC, like in TCP, is not fixed though there are recommended implementations. Furthermore, stream prioritization, something not present in TCP due to the single stream nature, is also not fixed. A implementation that offers prioritizing which streams to transmit data for by SRPT would likely see similar outcomes to Homa. I am not aware of any implementation that does so, but it is a freely configurable element of that protocol. I see nothing that would materially stop a comparable solution from being dropped in.
Mainly, unlike TCP, the design of QUIC has knobs in many of the right places so that you can reasonably configure the desired propertys with minimal design clashing.
Really, the main fundamental distinction in steady state operation is the resend mechanism being largely ack driven TCP and QUIC whereas Homa is largely nack driven.
I am of the opinion that ack driven is much superior technically, which is part of why I think Homa is a bad design, but likely for different reasons and considerations as to how it relates to transport protocol design than the designers of QUIC.
This is also not to say that I think QUIC is the best protocol, QUIC has a number of design and performance weaknesses, but it is overall a reasonable and material improvement over TCP and should be considered the standard to beat.
On a contiguous shared data area on the order of 1 GB, you are only tolerating a 2 MB overallocation on 1 GB of data area (~1/512 overhead), but saving ~2 MB of PTEs per process sharing the data (including the first) so you can come out ahead even if you are sharing at all.
At 300 GB you can even reasonably use 1 GB pages for a even more ridiculous ~1/128M overhead and effectively never run into the problem.
The official MsQuic benchmarks [2] tuned for the standard QUIC benchmark [3] with massively over-provisioned hardware does not even reach 8 Gbit/s on large sends with negligible RTT and loss.
If the highest performing public implementation can not even reach a paltry 8 Gbit/s it is fair to say they are all poorly performing.
As for what I would consider “okay” that would be around 30 Gbit/s per core, bottlenecking on decryption (assuming you do not use the allowed null encryption for benchmarking), or bottlenecking on ~1/3 of the bulk 1 Kbyte memcpy() throughput on your hardware, whatever comes first. “Good” is on the order of 100 Gbit/s per core or the same external limiting factors.
[1] https://www.sciencedirect.com/science/article/pii/S014036642...
[2] https://microsoft.github.io/msquic/
[3] https://datatracker.ietf.org/doc/html/draft-banks-quic-perfo...
And then in basically every other respect it is worse including compatibility.
That is a sender-driven resend and as such relies on the sender identifying the “send lost” condition. However, the entire protocol is designed around not doing that and thus you can only feasibly rely on “should have received a reply by now” as your timeout.
But Homa is intended to be a RPC protocol. So you send to server, wait for server to process command, then wait for server to send reply. Your timeout depends on the variable and heterogeneous command processing time.
Even if you were able to give a separate correctly tuned timeout for every possible RPC that is still awful. Any RPC with long processing time should not trigger the timeout until the expected reply time, but if that is far larger than the RTT then you are waiting a tremendous amount of time.
For instance, a select query that is only a few bytes long (and thus fits in one packet) could take seconds on a large database even though the database server is physically nearby and only microseconds away. In that case you would have to wait seconds before timing out instead of just microseconds like a ack-based design could achieve. The worst-case transport latency becomes application-specific instead of related to the application-independent transport parameters like RTT. That is troubling.
2. TCP can be forgiven for that given it predates asymmetric cryptography and even DES. Not considering it relevant in a new protocol this side of the millennium is much less reasonable.
4. MsQuic at 7.5 Gbit/s: https://microsoft.github.io/msquic/
However, software QUIC implementations are generally much slower than software TCP implementations. This is not due to any fundamental protocol design limitations, it is just that most/every QUIC implementation is poorly implemented for maximizing performance.
You can, of course, do multiple times more throughput than QUIC or TCP with a protocol better designed for performance, but that is likely orthogonal to your question.
1. No way to detect whole RPC loss. Since there is no outer connection state, if every packet in the send-side of a RPC is lost then there is no way for a server to detect that it should issue a resend. The RPC is just lost to the ether. This affects small messages, like messages that fit in a single packet, more since there are fewer packets in the send-side.
2. Related to the above, there is no builtin encryption support. So, if you want encryption then you need to layer it either above or below.
3. Benchmarked performance is awful. The 60 kB average message case in [2] Table 4 takes 5(!) hyperthreads to average 20 Gbit/s. That is just 4 Gbit/s per hyperthread. Even a totally naive one-packet per system call network protocol design and implementation should get to ~8 Gbit/s per hyperthread. 30 Gbit/s per hyperthread is easy with just a little focus on performance.
4. Despite all the performance design problems in QUIC (though still faster than Homa) it already solves basically every problem Homa is trying to solve in a much cleaner way. Stream IDs correspond to RPC IDs. Stream Max corresponds to Grants. Multiple streams under single Client allows prioritization.
Except you do not randomly lose entire messages. You can compact small messages into packets. You get more precise RTT time allowing more accurate pacing/congestion calculations. You get builtin encryption. It survives ossified middleboxs. It has multiple ack frames/packets reducing ack overhead.
The only real difference is that Homa uses explicit receiver Resend instead of implicit sender Resend. Except that actually consumes significantly more receiver resources in non-trivial loss scenarios especially due to Resend packets only supporting a single Resend span. It also incurs higher latency and has higher requirements on the entire lossy network path due to all the extra transiting data that you do not want to lose.
And those are just serious problems off the top of my head after reloading the RFC into my head. I can come up with some more if needed.
[1] https://github.com/johnousterhout/homa-rfc/blob/main/draft-o...
[2] https://www.usenix.org/system/files/atc21-ousterhout.pdf
They called it 'int' instead of 'int32_t' because it was meant to be the natural word size. It was supposed to float as the natural word size increased and allow code that worked on one word size to transparently work on the new word size.
Not floating to 64-bits on amd64 means their rationale for not giving it a fixed size was wrong. That is a simple width-doubling to another power of 2, the easiest possible case to expect to work transparently, on a compatible instruction set and they still chose to not float the size.
You can not just transparently change the word size and expect it to work, so you should just fix the size or use a fixed size. The notion of only having floating sizes as primitives was a mistake.
Memory copying is on the order of 100 gigabytes per second. You can do 80 full payload copys and still out-pace a dog-slow 10 gigabit per second connection. If your bottleneck is memory copying you are either doing something very wrong and doing way too many copys or congratulations you have implemented one of the fastest network stacks.
Supervisor calls are also very fast, on the order of 100 ns up to maybe 1 us with all the Spectre mitigations. Even if you did something as stupid as one supervisor call per packet, you would still be getting on the order of 10 Gbps at the long end there and 100 Gbps at the short end. Which, again, means congratulations are in order because you have implemented one of the fastest network stacks. If you do batching and add just 10 us (us, not ms) of latency then that entire cost is so small as to be irrelevant. You are going to bottleneck on your memory copying first.
Network stacks are so slow almost entirely due to poor protocol design and poor protocol implementation. Usually both.
Please enlighten me how that blurb supports the common reading of the claim: Your data, … secure. Note that and is a additive conjunction, so the components can be safely examined separately.
A regular person would assume that means your data is secure against tampering and disclosure. If you then say: “lol, jk we do not do anything related to securing your data” in the fine print then in a reasonable society you should be required to remove that more prominent false large print.
You can always go back and reword your large print to be more accurate without making deceptive claims to your benefit. Weird how the deception is always beneficial.