555 karma · joined June 1, 2014
Or an ISP can install libreqos, or preseem, or paraqum...
Ironically I ran tcp vegas and SFQ, by default, since roughly 2002, since I cared about voip quality in particular. The rest of the world, didn't. Thus fq_codel was born...
Most people do not measure in-stream latency, rather multiple streams, so, assuming you did or did not pick up fq_codel, a packet capture will show you the tcp RTTs being managed or not. Note that on a system where the tcp is originating and then transitioning htb on the same box, another subsystem (TSQ) managed the rates fairly well before hitting HTB.
Kathie nichols has also spoken about Codel multiple times.
My biggest gripe about that project is that everyone focuses on the ECN change, and nobody looks at the effects on normal everyday traffic, where fq_codel or cake utterly smoke the dual-pi AQM. There actually is L4S support in fq_codel now, but they are not testing it. :(
In general fq_codel is superior to pie in just about every way. There is a specific thing that pie does that makes it appealing to hardware designers is that it is O(1) on egress and fq_codel is not (but there are ways around that). fq_codel achieves less latency than pie does, and faster than it does. codel takes a bit longer than we would like.
fq-pie still has worse tcp latency (using the same fq algorithm as in fq_codel and cake), and cake solves all kinds of edge cases that no other fq+aqm+shaper can - per host FQ (solving torrent issues), ack-filtering, nat transparency, a better codel model, easy to use link layer compensation, diffserv - and my favorite feature is actually that it runs more or less the same as a default qdisc, as a shaped one. I had hoped it would replace fq_codel at both line rate and at the shaped cases, but it did prove a bit too cpu intensive on cheesy hardware and the edge cases not noticeable enough on simple benchmarks.
I would like to multi-thread it, and add some new features, discussion here: https://docs.google.com/document/d/1tTYBPeaRdCO9AGTGQCpoiuLO...
Delighted to see vyos pick it up!
Next is the shaping vs variable rate issue, on outbound and inbound. On outbound - Ideally the fq_codel or cake algorithms are located right on the bottleneck link, and respond to changes in the available bandwidth from the upstream dynamically (asserted by ethernet pause frames for example), and managed underneath by BQL (ethernet) or AQL (some wifi(ath10k,ath11k, mt76, mt79)) , though true native fq_codel support exists for the ath9k) which store up enough data only to service one interrupt. These algorithms are much (20x!) lighter weight than shaping to a fixed rate is, and like I said
Codel was designed to deal with variable rate links. A fixed shaping rate was not what we intended, but has become one of the most common and finicky use cases, dang it. It is the FQ that matters most in most cases, however.
On inbound shaping, it's a SWAG, where we recommend 85% or so of the provided rate as a starting point. You might need more, you might need less, due to the behaviors of "slow start" in particular, it is never going to be accurate enough, and we keep encouraging the ISPs merely to manage their own egress well enough (ideally dynamically and without a shaper) so inbound shaping is not needed. Libreqos/preseem/bequant/paraqum all make middleboxes that do ISP level shaping. Libreqos (for whom I work these days) has pushed CAKE to where an ISP can do 10k subscribers at 25Gbit at 50% of cpu on a cheap ($2k) Xeon or Ryzen box.
So you get something that works well on the rrul test... in the morning but not the evening. Folk then complain that their bandwidth from their provider is variable rate and shaping via their other box does not work all the time, and I go back to hoping more ISPs get told how fix it at, egress, truly right at their end.
There is another project which perhaps can be ported to vyos, called cake-autorate, which attempts to use active measurements to cope with variable rates.
Flow Queuing allows applications not creating queues to bypass those that are. It is mildly different from fair queuing: https://ieeexplore.ieee.org/document/8469111
Load shedding, at least for packets, benefits from head drop more than tail drop. Only the codel algorithm does this but codel has been applied to other forms of queuing like Uber dealing with a concertgoer overload.
Kathie Nichols has given some great presentations lately: https://www.understandinglatency.com/
There are a lot of unbounded queues in many applications & libraries, even in rust. They make me twitchy. Even with bounded queues the number chosen is usually arbitrary. I wish a timed queue was a default rather than a length.
I highly recommend Kleinrocks work on these subjects.
I am grumpy big vendors like juniper have yet to adopt smart queues... despite the obvious benefits. https://blog.cerowrt.org/post/juniper/
https://www.asce.org/career-growth/ethics/code-of-ethics
I wish more software folk did.
1) The data paths MUST prepend a timestamp to every packet 2) HW invsqrt for codel or cobalt AQM is 3k gates, checked on output 3) 32k 4 way set associative cache is a std IP block 4) HW Packet hasher - three different outputs
There are a few states in cake that are optional, but I remain boggled that step 1!! did not happen within 2 hardware releases of switch chips after van Jacobson and kathie Nichols published codel as a replacement for RED in 2012.
Example:
https://www.reddit.com/r/eero/comments/u7xm83/gen_2_sqm_vs_g...
The 8 way set associative hash is also much better than the direct map hash other things use, and I agree 256 flows is too small, 1024 set associative is great, 16k direct map comparable.
I wonder how well these would work on the GFW?
I rather hate the gross over-characterizations of SV as being VC-led (or destroyed). There are plenty of old fashioned network hippies still there, very concerned about the destruction of the formerly egalitarian ideals of the internet being ignored by the current ethos, people that still work on GPL'd and open source code, living in pain or poverty due to lack of credit, respect or support from the current regimes. The internet is a vast, shared resource that we hoped would benefit all personkind if we continued to understand the benefits connectivity brings to us all. I still remember and cherish the ethos that setup multiple non-profits to be its stewards - the hundreds of IXPs, in particular, but also isoc, arin and the other RiRs, places like isc.org, the internet archive, and wikipedia - all struggling to be heard over the latest grifter-driven-fad.
There is still room for entrepreneurship and innovation in SV and the world, but preferably greener and more self sustaining than it has been for the last decade.
A key attack vector identified in these documents was was "De-commoditize protocols".
I also wrote a now somewhat out of date cautionary note about all the pitfalls you can land into here: https://www.bufferbloat.net/projects/codel/wiki/Best_practic...
And over here, while we convinced systemd to adopt fq_codel by default, I cautioned against drawing conclusions from within the box simulations verses the general internet's behaviors: https://github.com/systemd/systemd/issues/9725#issuecomment-...
I am perversely glad measuring this stuff irks so many other people besides me!