What went wrong: SCTP (2009) | Hacker News Reader

What went wrong: SCTP (2009)

blog.ipspace.net

48 pointsmariuz44 comments

"However, by far the biggest showstopper to SCTP adoption is the lack of session layer in TCP/IP and the broken Socket API. If you want to use SCTP with the Socket API, you have to indicate the protocol to use in the socket call, which means that every application that would benefit from SCTP support must be changed, recompiled and tested. There is no way that you could take existing applications, add SCTP support in the operating system and have a better-performing Internet as the result."

Ah. "Ivan Pepelnjak, CCIE#1354 Emeritus, is an independent network architect." Ivan Pepelnjak also doesn't seem to understand network protocols. SCTP was never intended to be a TCP replacement or a basis for TCP or whatever the hell he thinks it to be. It is a separate protocol which is designed to provide a different set of features to a different set of applications.

I'm sorry that he and the large number of people like him, who would like to have exactly their use-case covered by exactly the simplest possible interface, preferably baked into the kernel or even better, the hardware, for the best performance. It's not going to happen.

I'm also sorry I have to live in a world where networking === TCP, over port 80 or 443. It wasn't supposed to be this way.

What went wrong with SCTP? By the time TCP-friendly congestion control research had reached a critical mass and started producing results, everyone had given up, adopted TCP as the sole transport protocol, and the deck was strongly stacked against anything that wasn't running on TCP.

I think I can credibly defend an argument that what went wrong with SCTP was that it got an IP protocol number rather than a UDP port. That one decision meant that, on most operating systems, using the protocol would require a kernel change, and virtually no middlebox would pass SCTP frames unmolested. They saved eight bytes and some negligible amount of checksum processing, and got no other benefits from being a first-class protocol.

You might be able to credibly defend it, but it's still depressingly silly that we're cut off from a large swath of our extensible-by-design protocol stack.

These days, people justify HTTP for everything with the argument that opening up a TCP socket is not guaranteed to work as reliably as HTTP-over-TCP.

They might be right, but god that sucks.

I see arguments like this as a form of the naturalist fallacy. Not every aspect of the original TCP/IP design is worth preserving and exploiting. We don't need an URG pointer. We don't need IP source routing (really, we don't need the IP options we have now). We don't need a "length" field in UDP. &c.

TCP should have been built on top of UDP, as should ICMP. The IP "protocol" field doesn't do us much good. I don't think we should fret too much about not exploiting it.

For the record, IP source routing has its use cases. It's done via firewall or high-end router policy, and it's not a federated protocol. but it's handy.

They got ECN capability, which would have required OS changes even if it was built atop UDP.

There's a spec for SCTP over UDP and libraries that implement it, so what's stopping anybody now?

In fact, WebRTC's datachannel uses it

Nothing, but there hasn't always been that spec, and SCTP over UDP is not what people usually mean when they refer to SCTP.

Because if you're going to start building a protocol on UDP from scratch today, you can do much better than SCTP based on what concerns you're trying to solve. Pretty much every new UDP protocol pays at least lip service to the mobile use-case, and tries to get to zero roundtrip session establishment for majority of sessions ( and here the 4-way handshake of SCTP sticks out like a sore thumb).

But then there are some people who might be more interested in making a secure protocol from scratch (rejecting the layering complexity and third-party dependence of SSL). For instance: http://cr.yp.to/tcpip/minimalt-20130522.pdf

You are possibly right, although I suspect the kernel change issue is less of a problem than the second. Getting middleboxes to pass UDP unmolested....

You can at least configure virtually every middlebox to pass UDP. You cannot reliably do that for random ipprotos.

SCTP should have been promoted and is actually pretty nice on paper. Every time I see someone re-implement multi-plexing and datagram/message parsing out of a TCP stream I think of SCTP. Every time I see custom 'reliable delivery on top of UDP' I think of SCTP. We might not have had QUIC or HTTP/2 in the current form if SCTP was ubiquitous.

The big problem is that that promotion really needs to be aimed at a highly specific community: operating system and network equipment vendors.

The original article was written 6 years ago and wasn't a novel position at the time. It's hard to say anything's changed since then because app developers are still blocked waiting on mainstream middle-box support and I don't see that really happening without some sort of pressure on the vendors.

Relevant Operating System community news here: https://news.ycombinator.com/item?id=9797932

SCTP support has just been removed from DragonFly. The code had been present for 15 years (inherited from FreeBSD) and in these 15 years there wasn't even a single known user.

If there really are people using it it's effectively in a very very small niche and they will have to do everything themselves. There is no way generic operating system developers can work on this protocol.

The only way I see this cycle changing is if someone ships a real application which uses SCTP to provide significant benefits to users. If deploying SCTP improved performance or reliability for video-chat, a popular game, etc. there'd be a more compelling reason to go to e.g. Cisco, NetGear, etc. and ask them to ship kernel-level code with likely security impact to everything they sell.

WebRTC is the most obvious candidate for this but maybe one of the HTTP2-over-SCTP experiments will go well enough that someone will ship it in a real browser and start the process of saying “Web users get n% performance improvements if SCTP is implemented”. That certainly seems like a rather long game to play, however.

I'd like to also add to this that SCTP doesn't ship on Mac OS X either by default. Although, in recent years there have been development on an SCTP over UDP standard, allowing for user-mode SCTP. Yet, I still haven't seen a mature version of this API ship on Mac OS X.

Some time ago, I also did performance testing on SCTP vs. TCP. It's surprisingly sad how much worse SCTP's performance was compared to TCP. I imagine that this is because just isn't as much work put into SCTP as TCP.

SCTP uses 32 bit checksums rather than 16. It may be the case that your hardware can do 16 bit checksums in hardware, but for SCTP it has to do the checksum in software. (I think it used to be the case and maybe still is that SCTP vs TCP benchmarks would disable hardware checksums so that the results weren't biased unfairly in favor of TCP.)

even at gigabit speeds, unless the implementation is really, really bad, checksumming will absolutely not be the issue here as far as my experience goes.

Benchmarks should be consulted before preformance issues are debated.

I once made a CRC64 implementation, and single core performance was at 1150 MBytes/s on Java 8. So, yeah.

Some Intel NICs (at least the 10G ones, but some Googling suggests some 1G as well) support SCTP checksum offload. Also, the checksum is a CRC32C and the SSE4.2 CRC32 instruction can calculate these really fast.

WebRTC DataChannels are actually built on SCTP.

Yeah I was going to mention this. I presume since no OSes natively support SCTP out of the box they just emulate it on top of UDP.

Interestingly, Emscripten emulates UDP by using SCTP's unreliable data packet mode with max retransmission set to zero. Relatedly, there is an excellent library for game networking called enet, which implements reliable and unreliable transmission. Enet is included in Emscripten.

So if you use Enet in emscripten you have: Enet, which uses UDP, which is emulated via SCTP, which is transported via UDP! It's pretty crazy but it actually works well.

They do and don't. They use a user-space implementation based on UDP.

With datagram TLS in between, which added encryption. Which is preferable to what you'd do without UDP encapsulation.

Is there something wrong with RFC 3436?

So to summarise, nobody uses it because nobody uses it? I don't buy it.

I think TCP is "good enough" for most cases, so people use it. SCTP is great, but TCP isn't broken. Worse is better and all that.

This writer seems to think the killer problem SCTP solves is end-system level IP multihoming. I think that's actually not a problem that many apps really have.

To me, the real promise of SCTP always seemed to be that it was a reliable out-of-order datagram protocol that played nicely with TCP. Essentially: that it killed the need for applications that can't deal with the slowness of in-order delivery in TCP to write their own protocols.

I think the big mistake with SCTP was in making it an IP protocol, rather than a protocol that lived underneath UDP.

It's a problem that apps have, but they just shift it onto the user.

That's a bit hard for me to fathom, because the overwhelming vast majority of users don't have multiple Internet connections, and serverside applications can make use of multiple Internet connections without protocol-level multihoming.

> because the overwhelming vast majority of users don't have multiple Internet connections

your shiny iphone/andriod etc. are all multihomed...

So, the problem is not shifted onto the user?

Correct, it's not.

It's shifted on the application developer, who is the "user" of the network stack.

Next time you reconnect your SSH session after changing from one WiFi network to another one, consider that this is a degenerate case of multihoming.

ummm.... mobile devices, though not truly multihomed (in the sense of using multiple NIC's simultaneously) approximate multihoming in a time-division sense.

Well "nobody" is a bit misnomer. It's still widely used in telco networks, transporting control messages to/from your mobile device within the core network, and in interconnects between carriers.

Aren't you basically saying the same thing? TCP was good enough for most cases, so nobody uses it. And anybody who wants to use it can't, because it isn't widely enough supported to be reliable over the public internet. And it isn't widely supported because nobody uses it.

MCTCP (which probably is most famous right now for being used by Apple for Siri) is a cool but complex hack on top of TCP to provide multi-homing without confusing middle-boxes. HTTP2/SPDY/QUIC have their own mechanisms to get multiple streams in one connection over TCP/UDP.

I think the multi-homing feature of SCTP is not so useful. It adds a layer of complexity without so much gain.What do you think of it.

1) It makes the SCTP header complex. Without it, IP address in IP header is sufficient. With it, IP addresses have to appear in SCTP header too.

2) Multi-homing makes it difficult to configure firewall rules, for example NAT(Network Address Translation). Linux tools like iptables, LVS(Linux Virtual Server) do not work well with it.

3) I think the benefit of multi-homing can be gained in Network-layer or Data-Link layer instead of transport layer, for example network interface bonding, etc. It relieves the complexity of the application in this way.

I must iterate:" I’m taking a 3000-foot perspective here, it is still a far away perspective although less a 0 than the author, and whatever I’m writing could be completely wrong. If that’s the case, please point out my mistakes in your comments."

Is the multihoming of SCTP useful in comparison to IPv6 multihoming?

Has anyone implemented multihoming IPv6 capable of creating a continuous SSH session when switching networks?

SCTP is in Sun/Oracle Java SE 7. Examples show the multi stream aspect. Can't speak for multi-homing. Of course Java is only popular on one end of potential connections.

Hmm, 2009...

Reply on news.ycombinator.com