Comment by choocroot | Hacker News Reader

Comment by choocroot | Hacker News Reader

« Note the download isn’t going to hover exactly at 1.0K/sec — > the actual download speed as reported by wget is an average over time. In short, you’ll see numbers closer to an even 1.0K/sec the longer the transfer. In this example, I didn’t wait to download an entire 4.2GB file, so the 10.5K/s you see above is just wget averaging the transfer speed over the short time I left wget running. »

Wrong: you see 10KB/s download speed because you are not throttling the incoming packets but the outgoing packets!

So, what you are really doing is rate limiting outgoing ACK packets to 1KB/s, which delays your outgoing ACKs, hence preventing the remote server from sending you data at full throttle.

You can see that with a tool like "bwm-ng" to see Rx/Tx speed, and you'll see exactly 1KB/s on Tx, and something variable on Rx.

The incoming rate will fluctuates depending on the TCP window setting used by the server which translate to how many non-acknowledged packets are allowed to be sent by the server.

> Wrong: you see 10KB/s download speed because you are not throttling the incoming packets but the outgoing packets!

Yep. TC's default is to policy outgoing traffic, which in OP's example is a bunch of TCP ACKs essentially. Instead, they should be using ingress keyword, something like described here:

http://blog.stevedoria.net/20050906/ingress-policing-with-li...

Caveat emptor: ingress rate-limiting is hard. Long story short, it all boils down to what you do with non-confirming packets: There are two alternatives, and both are rather sub-optimal. You can either buffer/delay packets in kernel space (default, which leads to bufferbload and memory waste), or drop (which author linked above opted for, which leads to excessive retransmits and bandwidth waste).

The drop vs buffer decision is no harder for outgoing or incoming packets. In either case: if you're trying to simulate a different kind of network, do what that network does. If you're just trying to get good QoS on your gateway router, then use a smart AQM that will buffer only to the extent that is reasonable, and then drop or ECN mark when buffering threatens to add too much latency.

What actually happens when Internet traffic makes the jump from my ISPs big-pipe backbone connection to my much slower last-mile connection? They clearly can't cache the world, and "excessive retransmits and bandwidth waste" doesn't seem to be the case?

They drop your packets, both for ingoing and outgoing overflow.

But how is that different from the proposed method of rate limited which causes "excessive retransmits and bandwidth waste"?

Not being pedantic, genuinely trying to understand how this stuff works.

Middle ground between dropping [1] and buffering is Active Queue Management as wtallis pointed out above. State of the art AQM on Linux nowadays is CoDel [2].

Big network gear vendors that ship to ISPs have adopted early forms of AQM (both RED [3] and proprietary algorithms) quite a while ago - they had to:

(a) backbone routers have much smaller buffer-to-bandwidth ratio (compared to a PC or even home router), so endless buffering is not an option;

(b) buffer tail drops (i.e. what drop keyword does when added to tc filters/disciplines) interact really poorly with TCP bandwidth control algorithms, and rather badly with RTP streams, too (so drop is not an option either - it ruins user's connections; and

(c) ISPs and carriers would typically run links (on average) much closer to saturation than your typical home router, so situation where router has to make buffer/drop decision is much, much more frequent.

[1] My point above was that you don't really want to drop packets on the receiver side, after that packet already traversed expensive part of the network.

[2] https://en.wikipedia.org/wiki/CoDel

[3] https://en.wikipedia.org/wiki/Random_early_detection

It's common to find that the buffers in front of the bottleneck last-mile link are sized for the highest tier of service regardless of what speed you're actually subscribed for. DSLReports has the only browser-based speed test that measures bufferbloat, and they have extensive data from that testing: https://www.dslreports.com/speedtest/results/bufferbloat?up=...

Your modem and the box on the other end of that bottleneck link are probably buffering far more than is reasonable. There's simply no reason for a cable modem to ever have in excess of 1s worth of backlog.

Reply on news.ycombinator.com