Hacker News Reader

"what forces have made TCP the only transport protocol of note for so long?"

Well, another one is that rather a lot of things block anything that isn't TCP, DNS, UDP, and anything else that isn't the bare minimum for "the web" to work. They'll even go so far as to pick out which types of ICMP messages to pass through.

The internet at large has been sitting on SCTP for a very long time, because nobody expects SCTP to be able to get through firewalls. (On that note, perhaps the solution is to reformulate SCTP in terms of UDP, as that's what everybody else seems to be doing.)

UDP is quickly becoming the base protocol for "I don't want TCP but I do need something that gets through firewalls". Sort of like how so many things float on port 80 or HTTP that don't really need to, but it's what can get through firewalls and such.

UDP is quickly becoming the base protocol for "I don't want TCP but I do need something that gets through firewalls".

Exactly right. No need to invent a new protocol number in the IP headers. UDP is the perfect vehicle for custom protocols. And once you start experimenting with a somewhat intelligent custom protocol, it's amazing how easy it is to beat even well tuned TCP stacks on average over a large number of users. This is especially true, by large large margins, over mobile networks.

Why does window scaling need to be OS-global? That would imply that window scaling will also fail when you have several machines on the same network, since they won't be doing anything special to collaborate on window sizing.

Conversely, any hints and clues to window sizes that are provided by network packets can be read from a user process and so don't need to reside in the kernel either.

If multiple processes running on a single physical machine can't avoid this "window flapping" problem without cooperating, then multiple unikernels running in VMs on a single physical machine also can't. VMs are not magic.

Rather, the need for cooperation actually extends to groups of physical machines sharing a network connection too. Indeed, that's the key insight underlying objectivist mouse-based congestion control:

http://web.mit.edu/remy/

I might not be reading this correctly, but in the absurd extreme, you can keep extending this argument farther and farther out beyond a single machine to a LAN and onto the wider internet. In that version of the argument, all the nodes on the internet need to have the global view of network state on every other node.

The controlling principle behind almost all congestion control algorithms is that a single node can derive the properties of the path between itself and the other end by using the state of the transfer visible only on that one node. Once you reject that principle, any algorithm you come up with will quickly become unscalable.

http://en.wikipedia.org/wiki/TCP_window_scale_option

Can you expand on why "window scaling needs to be OS-global to avoid flapping". I have googled it and a I got to

http://en.wikipedia.org/wiki/Route_flapping

and I couldn't work out how this related to TCP living in the Kernel.

Not trolling, real question. Thanks in advance.

My guess is that the OS kernel has a better idea of the total bandwidth available than each individual process. If two processes decide to increase their bandwidth at the same time, and both of them would miss a packet falling back. I'm guessing the flapping in this case is processes increasing and decreasing their bandwidth because of no central control of the data rates.

Well, I suppose he is talking about how the kernel schedule tasks.

TCP windows is strongly related to time and buffer capacity and consumption.

If you don't control the time, you can't control tcp-windows. I suppose.

the TCP stack is controlled entirely by the kernel. so it handles things like requesting dropped packets and rate limiting.(which relates directly to "packets in flight")

with UDP, that layer of ACK, ordering and rate limiting is gone.

if you want to send a million packets a second to 56k modem, sure the kernel isnt going to stop you.

Whilst it might be possible to have per TCP connection windowing parameters, I doubt it'd be very efficient.

> Whilst it might be possible to have per TCP connection windowing parameters, I doubt it'd be very efficient.

If you had kept going here and explained why you doubt that, you would have answered the parent's post.

So how does QUIC solve the problems you mention? Afaik it does not require any kernel-level code.

That was my first question too. QUIC is all user-land code.

While, again, this is a fun spin on TCP - QUIC rides on UDP. Interestingly enough I pulled the stats from my home network for the past 30 days and I had roughly 300MB and 9k QUIC sessions (Chrome defaults to QUIC for the majority of Google services) via my firewall. If you want to see QUIC in action pop this into your Chrome bar with a Google endpoint open: chrome://net-internals/#quic

Compared to HTTP things like QUIC make perfect sense - especially when you consider all the RTT wasted on pipelined connections to the same site. You're only opening all those sockets because they're not multiplexed in any manner - your browser wants a JPG, grab the JPG. Your browser wants another, do it again...

So while this is light hearted, there's some validity to next gen web protocols that are more adapted to how the data is consumed.

Multiplexing is in SPDY -> HTTP/2. QUIC shouldn't be "compared to HTTP" – it should be compared to TCP+TLS. http://blog.chromium.org/2013/06/experimenting-with-quic.htm...

You should read the design document and specicifaction rationale, it has more details than a blog post. Considering on the first page, in bold font it describes QUIC as "Multiplexed Stream Transport over UDP" it should only be compared to TCP+TLS in operation - but it helps to solve the multiplexing problem for anything that decides to take advantage of it directly. The majority of applications using TCP+TLS as a transport today have the same problem and open many sockets during operation.

Agreed SPDY and HTTP/2 have multiplexing built in - but so does QUIC.

From the design doc: https://docs.google.com/document/d/1RNHkx_VvKWyWg6Lr8SZ-saqs...

"This is a working document, for group discussion and editing, which we expect to evolve into a somewhat fleshed out design document. The expectation is that we will flesh out a design for a tunneling protocol, running atop UDP, which can multiplex a large number of streams between two endpoints (a client, which initiates the overall connection, and a server). Each stream may, for example, be nearly equivalent to an independent TCP connection."

Edit:

And this...

"Pairs of IP addresses and sockets are finite resources. Today, too many distinct connections are routinely made between clients and servers, utilizing a multitude of sockets, and often carrying redundant information. A multiplexed transport has the potential for unifying the traffic and reducing the port utilization count. A multiplexed transport can unify reporting and responses to channel characteristics (packet loss, etc.), and also allow higher level application protocols (such as SPDY) to reduce or compress redundant data transmissions (such as headers). In the presence of layer-3 load-balancers, a multiplexed transport has the potential to allow the different data flows, coming and going to a client, to be served on a single server."

it doesn't. QUIC is a slow inefficient toy protocol. mainly as a tunnel for accepting spdy, a protocol "designed" with TCP in mind.

http://www.connectify.me/blog/taking-google-quic-for-a-test-...

things might have improved, but over slow lossy links, its really not worth the hassle.

the OC alludes to the point that UDP requires less kernel intervention, which means you can do more funky things.

I thought the main benefit is supposed to be no head of line blocking in multiplexed streams.

"Well, in TCP, window scaling needs to be OS-global to avoid flapping. This means TCP lives in the kernel. And this means, if you want a protocol with any property similar to window scaling, it also needs to live in the kernel."

Sorry but this is a huge assertion with absolutely no supporting evidence or argument. What exactly do you mean by "flapping"? Why do we need to avoid it? And why does a protocol need to live in the kernel to avoid flapping?

I suspect those assertions are based on an assumption that congestion control gets done in a cooperative manner across connections. (I'm assuming the "window" here is congestion window.) I've never seen or heard of a TCP stack that somehow takes into account all TCP flows in a system. As a matter of fact, even if that was the case, it would be too limited a view of the responsibilities of TCP congestion control.

The congestion control algorithms are working on a global picture of congestion... as in across the entire network path between the two communicating nodes. Various cc algorithms (reno, vegas, bic, cubic etc.) are using their own techniques to probe the network path between the two endpoints (either using packet drops or round-trip delay increases as a signal of congestion). As such, the concern of congestion control extends beyond just the one node where the algorithm happens to be running. But the important point here is that each node on the network uses only the signals available locally at that node. And for the purposes of this argument, you can consider each of the TCP endpoints on a single machines as an independent node.

I don't see any convincing case why two separate instances of these state machines can't be running independently in kernel and user space. Since ultimately, if they're competing for bandwidth over a common, narrow part of the pipe, they'd each use their own congestion signals to arrive at the same conclusion (assuming the protocols are designed for fairness).

> I've never seen or heard of a TCP stack that somehow takes into account all TCP flows in a system.

Taking into account all flows is indeed not useful. But there's some useful middle ground between "all flows" and "each flow completely separately".

At work we've got a transparent proxy with an integrated userspace TCP stack. When it's installed in a mobile operator's network, all of that operator's subscriber traffic will be going through it. Now, for each subscriber all the flows between the terminal and the proxy will be in the same "congestion domain"; they're guaranteed to be competing over the same network resources (whether it's backhaul or radio capacity). And in fact for the scarcest resource of the radio capacity each subscriber is essentially in their own congestion domain due to the way the radio base stations scheduling algorithms work. Having the custom TCP stack treat all of a single user's flows as a single unit rather than individually is a big win.

(The original claim that TCP must in the kernel for "window scaling" doesn't make any sense to me, no idea of what it could be referring to).

Hmm... interesting insight from the core network. I guess another reason to use a UDP (encrypted) protocol would be to prevent such middle-box shenanigans ;-)

>what forces have made TCP the only transport protocol of note for so long?

It is simple to use and good enough for most purposes. Streams and games for a long time already.