Hacker News Reader

Comment by cesarb | Hacker News Reader

> Like why not just set all new hardware to an MTU of 9000 and wait ten years?

The hardware in question is Ethernet NICs. However, for you to set the MTU on an Ethernet NIC to 9000, every device on the same Ethernet network (at least the same Ethernet VLAN), including all other NICs and switches, including ones which aren't connected yet, must also support and be configured for that MTU. And this also means you cannot use WiFi on that Ethernet network (since, at least last time I looked, WiFi cannot use a MTU that large).

View on HN

Sending a jumbo frame down a line that has hardware that doesn’t support jumbo frames somewhere along the way does not mean the packet gets dropped. The NIC that would send the jumbo frame fragments the packet down to the lower MTU. So what’s the performance impact of that fragmentation? If it isn’t higher than the difference in bandwidth overhead from headers of 9000 MTU traffic vs. 1500 MTU traffic then why not transition to 9000 MTU?

But how does the NIC know that, 11 hops away, there is a layer 2 device, which cannot communicate with the NIC(switches do not typically have the ability to communicate directly with the devices generating the packets), that only supports a 1500 byte frame?

Now you need Path MTU discovery, which as the article indicates, has its own set of issues. (Overhead from trial and error, ICMP being blocked due to security concerns, etc...)

If you block ICMP you deserve what you get. Don't do this. (Edit: don't block ICMP)

So now you're trying to communicate from your home machine to some random host on the internet (website, VPS, streaming service), and you're configured for MTU 9000, the remote service is also configured for MTU 9000, but some transit provider in the middle is not, and they've disabled ICMP for $reasons.

They blocked ICMP, do you deserve what you get?

Transit providers should push packets and generally do. With PMTU failures it's usually clueless network admins on firewalls nearer endpoints. And no, you don't and I wish the admin responsible could feel your pain.

> Transit providers should

Agreed

> and generally do

Agreed.

Now if you can make it 'will always just push packets', we'll be golden.

Unfortunately, there are enough ATM/MPLS/SONET/etc networks being run by people who no longer understand what they're doing, that we're never going to get there.

To make matters more entertaining, IPv6 depends on icmp6 even more.

Why should it need to? Ethernet is designed to have non-deterministic paths (except in cases of automotive, industrial, and time sensitive networks). If you get to a hop that doesn’t support jumbo frames then break it into smaller frames and send them individually. The higher layers don’t care if the data comes in one frame or ten.

> Sending a jumbo frame down a line that has hardware that doesn’t support jumbo frames somewhere along the way does not mean the packet gets dropped

Almost all IP packets on the internet at large have the 'do not fragment' flag set. IP defragmentation performance ranges from pretty bad to an easy DDoS vector, so a lot of high traffic hosts drop fragments without processing them.

If we had truncation (with a flag) instead of fragmentation, that might have been usable, because the endpoints could determine in-band the max size datagram and communicate it and use that; but that's not what we have.

AFAIK, Ethernet has no support for fragmentation; I've never seen, in the Ethernet standards I've read (though I might have missed it), a field saying "this is a fragment of a larger frame". There's fragmentation in the IP layer, but it needs: (a) that the frame contains an IP packet; (b) that the IP packet can be fragmented (no "don't fragment" on IPv4, or a special header on IPv6); (c) that the sending host knows the receiving host's MTU; (d) that it's not a broadcast or multicast packet (which have no singular "receiving host").

You can have working fragmentation if you have two separate Ethernet segments, one for 1500 and the other for 9000, connected by an IP router; the cost (assuming no broken firewalls blocking the necessary ICMP packets, which sadly is still too common) is that the initial transmission will be resent since most modern IP stacks set the "don't fragment" bit (or don't include the extra header for IPv6 fragmentation).

It does mean packets sent to another local, non-routed, non-jumbo-frame interface would get lost. So you could, for example, maybe talk to the internet, but you couldn't print anything to the printer down the hall.

Fragmentation/reassembly is an l3 concept and not guaranteed to large MTUs when it is there.

PMTU-D will save their ass in some cases. But it's not safe to assume all routers in the path will respond to ICMP.

It doesn't matter that the routers respond to ICMP, it matters that they generate them, and that they're addressed properly, and that intermediate routers don't drop them.

Some routers will generate the ICMPs, but are rate limited, and the underlying poor configuration means that the rate limits are hit continously and most connections are effectively in a path mtu blackhole.

>It doesn't matter that the routers respond to ICMP, it matters that they generate them, and that they're addressed properly, and that intermediate routers don't drop them.

>Some routers will generate the ICMPs, but are rate limited, and the underlying poor configuration means that the rate limits are hit continously and most connections are effectively in a path mtu blackhole.

Sure. But I'm not about to sit here and name all the different reasons for folks. And since most here do not have a strong networking background running consumer grade routers at home, it seemed most applicable.

I could have used a more encompassing term like PMTU-D blackhole, but I didn't.

The worst case I just recently encountered with Jumbo Frames was with NetworkManager trying to follow local DNS server's advertised MTU but when the local interface doesn't support Jumbo Frames it just dies and keeps looping.

Even if you really want devices to use JF, some fail miserably because it's just not well thought out.

Reply on news.ycombinator.com