Hacker News Reader

Comment by Animats | Hacker News Reader

OSI's answer to TCP, TP4, was implemented in Windows 2000.

TP4 assumed a reasonably reliable data path, probably an X.25 virtual circuit, underneath. It didn't have all the timing, reordering, retransmit, and congestion control machinery that lets TCP work over bad links.

On the other hand, the OSI crowd had, at the time, more experience with scaling and routing. IP's class A, B and C networks, autonomous system numbers, and Border Gateway Protocol didn't scale well. Routing in the Internet is still something of a hack. Today's routers almost have to have an in-memory table mapping all 4 billion IPv4 addresses to Autonomous System Numbers. If memory wasn't so cheap, we'd be having real problems.

View on HN

TCP is in fact pretty shitty over "bad links". Standard TCP congestion control algorithms are designed to assume that packet loss indicates congestion; not link degradation. This results in actual "bad links" tricking TCP into thinking there's congestion... so it backs off (which doesn't fix the problem); backs off some more (which still doesn't fix the problem); etc.; resulting in terrible throughput in the face of packet loss.

There are alternative congestion control algorithms that mitigate this somewhat; but the best solution (which AFAIK there is no widely-used standard for) is to use forward-error-correction. But since the Internet, for the most part, doesn't actually have "bad links", this is not generally necessary.

Re: "Today's routers almost have to have an in-memory table mapping all 4 billion IPv4 addresses to Autonomous System Numbers."

Full BGP Table is around 500,000 entries, so you are off by several orders of magnitude there.

TP4, where the 4 stands for class 4, certainly had reordering, retransmission and congestion controll.

Other classes you could chose had varying degree of error detection and recovery. It also had a connectionless mode.

"You could choose". Therein lies the problem. TCP is automatic, which is a big win. You don't have to set parameters for TCP depending on the data path.

On the flip side, TCP forces you into a single ordered stream and head-of-line blocking. If you need a reliable yet unordered message stream, you have to reinvent the wheel on top of UDP.

Or use an internal work queue instead of buffering in the kernel and/or network; or use multiple TCP connections; or use a single SCTP (or SCTP/UDP) connection.

Any of these is less work and less bug-prone than "reinventing the wheel on top of UDP".

SCTP would be the ideal solution, but it's essentially broken on the wide Internet since virtually no NAT boxes understand it.

Using a large number of TCP connections breaks some of the assumptions behind TCP congestion control; this is one of the reasons why the SPDY/HTTP2 guys decided to layer a multi-stream protocol on top of a single TCP connection - and it still falls prey to head-of-line blocking.

Not sure what you mean by using an internal work queue - short of running a custom TCP/IP stack in kernel space or using some non-standard socket API, there's no way to work around TCP's ordered delivery.

Given these constraints, I can understand why designing a transport protocol on top of UDP might be reasonable. There probably is no need to roll your own, though - just use SCTP over UDP [1]. (Edit: I only noticed you did mention SCTP/UDP after I wrote my reply, sorry!)

[1] https://tools.ietf.org/html/rfc6951

SCTP would be the ideal solution, but it's essentially broken on the wide Internet since virtually no NAT boxes understand it.

Hence my suggestion of SCTP over UDP.

Using a large number of TCP connections breaks some of the assumptions behind TCP congestion control

Which assumptions? Using multiple connections is pretty standard in the enterprise storage world.

My bad, I missed that you were referring to lost packets blocking future packets (as opposed to slow-to-process work units blocking future work units). You're right that this is very much a problem with a single TCP stream. (Multiple TCP streams help with this.)

> Which assumptions? Using multiple connections is pretty standard in the enterprise storage world.

TCP congestion control is designed such that, under steady-state conditions, each TCP stream traversing a physical link gets an equal share of the available bandwidth. When an application that uses multiple TCP connections competes with another that uses a single one over the same link, the resulting bandwidth distribution is unfair to the latter. (Edit: there's also another problem, related to bufferbloat, but that one is a lot trickier to explain).

Of course, this may not be a problem when used on an internal network (which I assume is the scenario you are referring to). On the public Internet, however, HTTP is becoming increasingly disruptive to other kinds of traffic due to the increasing number of parallel connections used by web browsers.

Ah yes. I'd say that's more a problem with TCP in general than in using multiple connections. The assumption that "1 TCP connection == 1 share of bandwidth" is at best a useful first approximation. I don't know that N TCP streams eating N times their "fair share" is really any different a problem than some-important-interactive-application being given equal bandwidth with some-irrelevant-background-download. (Though it might be a worse problem.)

I'd love to live to see the day that something actually better than TCP (that addresses these and other issues) dethrones it, but given how long IPv6 took to gain traction, I wouldn't be surprised if it didn't happen in my lifetime.

I think it's a wonderful thing that the TCP/IP model (as originally envisioned) does not preclude endpoints from implementing their own transport protocols on top of IP, irrespective of what the rest of the network uses.

Unfortunately, reality has conspired against this possibility in two ways: NAT boxes, and the fact that transport protocols are implemented in the kernel on most operating systems (hence, an application cannot use its own transport protocol easily).

The deployment of IPv6 is a real chance to, at least, fix the first of these problems. Let's hope it does.

Why is that a problem ?

You specify a parameter when you start, telling which variant to use. It's really not that different sitting down before you start coding a program and deciding whether to use TCP, UDP, DCCP or similar.

Because you don't know whether the end user running the program has a gigabit server-quality pipe or a flaky mobile connection.

Sure, but how is that different from the TCP/IP world ? You select the protocol based on what your application needs.

If memory wasn't so cheap, we would have transitioned to IPv6 long ago instead of coping with an ever increasing global routing table.

IANA and the RIRs are stepping away from their requirement that everyone route through their ISP's parent space. They quickly discovered there are a lot of reasons why Provider Independent Address Space (PIAS) is critical to any real business, and that nobody is going to re-number when they change ISPs (not to mention the hassle of having to deal with multi-site environments). The entire concept was a pipe dream.

If they hadn't yielded on PIAS then every enterprise network engineer out there was just going to deploy in RFC 4193 ULA space, say screw you to the IETF and implement RFC 6296 - NPTv6 on their perimeter.

Net-Net - Once IPv6 starts to get traction, we're going to have routing tables of roughly the same size, but they'll just consist of /48s instead of /20s.

Deploying RFC 4193 at least would not pollute the global routing table with lots of short prefixes. Those enterprise networks would pay for that flexibility themselves instead of forcing the cost onto the whole Internet.

Routers have gotten much, much faster, and memory has gotten much, much bigger. And a lot of the Longest Prefix Match stuff is highly optimized/fast switched anyways now.

Hence my inital premise: if memory hadn't gotten so cheap, we would have transitioned into IPv6 already. For all the good Moore's Law brought us, it is also the reason why we seem to be stuck in this NAT world with no end in sight.

The original belief with IPv6 is that End Users would route through their ISPs address space, and the IPv6 made renumbering so easy, that it would be simple for end-users to change providers and renumber.

This turns out not to be the case.

IPv6 will not reduce the size of the routing tables. Anybody with prefixes, and PIAS right now, will likewise get a /48 of IPv6 space and start advertising it into BGP.

If anything the more easily available space may result in bigger routing tables as a result. (Not everyone was able to get a /20 - particularly outside of ARIN space. For example - IPv4 addresses come very dear in places like Dubai.)

And NAT doesn't go away with IPv6. That was another belief that did not survive its first encounter with reality. It's simply called "Network Prefix Translation" now (NAT by another name)

> And NAT doesn't go away with IPv6. That was another belief that did not survive its first encounter with reality.

If it doesn't go away, it will be only because of ISP greed (i.e., charging customers for a larger number of publicly routable addresses). Otherwise, there simply is no reason not to give every customer a /56 or even a /48, given that routers won't be able to route prefixes longer than that anyway.

Fortunately, most ISPs offering native IPv6 so far have turned out not to be that greedy.

It has nothing to do with ISP greed. There is no shortage of IPv6 addresses, and, ISPs have every motivation to encourage their customers to use their IPv6 space.

The reason NAT doesn't go away is that customers don't want to use their ISPs IPv6 space, they want their own provider independent space.

The easiest way to do that, is to address all your devices from RFC 4193 ULA space, and then, on your perimeter, do RFC 6296 Network Prefix Translation to the ISPs IPv6 space.

Then, when you change ISPs, you simply make a single change on your perimeter rather than having to renumber hundreds of internal devices, DNS, configurations, etc...

I understand the argument for enterprise networks. Residential customers have everything to gain from using their IPv6 space, and provider-independence isn't an issue for them.

It's worth noting that NPTv6 is a lot better than NAT, though. For one thing, it operates strictly at the IP level and doesn't muck about with the transport layer. So, for instance, if you want to use a transport other than TCP or UDP, it should be possible - at least in principle - even if you are NPTing to and from ULA space. That's something you haven't been effectively able to do on the Internet for the past 20 years.

Totally agree with you across the board.

Reply on news.ycombinator.com