HNHacker News
TopNewBestAskShowJobs

dtaht

555 karma · joined June 1, 2014

submissionscomments
dtaht··on Rivers Cuomo is an active developer on GitHub
There are a lot of programmers that dabble as musicians, and vice versa. Licklidder was a pretty good guitarist, I am told.
dtaht··on Flent: The FLExible Network Tester
Switching to a router that supports SQM (ideally, CAKE) on the wan interface is the suggested solution. fq_codel on the wifi, helps a lot too, but first up is fixing the ISP connection.

Or an ISP can install libreqos, or preseem, or paraqum...

Ironically I ran tcp vegas and SFQ, by default, since roughly 2002, since I cared about voip quality in particular. The rest of the world, didn't. Thus fq_codel was born...

dtaht··on I moved my blog from IPFS to a server
Clever. I care about congestion control issues mainly. Got that handled? Tried ecn?
dtaht··on Ingenuity had more computing power than all NASA deep space missions combined
I have long advocated that we attempt dozens of "Deep Impact" style missions against the NEOs - short mission lifetime, perhaps months from launch to impact + a high launch rate - would bring the cycle time from innovation to exploration way, way down. The "Pluto Express" abuse of the word "express" rankled. Not having to fund a team for decades per single mission would be a savings also. If we shift our foci to closer targets, outside LEO, by delta-v and flight time, we will evolve our spaceflight technologies much faster. The moon is a lousy target, because landing on it, and returning from it is so hard, and Mars is too far.
dtaht··on Ingenuity had more computing power than all NASA deep space missions combined
Heh. I have long advocated that we attempt dozens of "Deep Impact" style missions against the NEOs - short mission lifetime, perhaps months from launch to impact, a high launch rate - would bring the cycle time from innovation to exploration way, way down.
dtaht··on Flent: The FLExible Network Tester
I have been steering folk at this new test - called crusader - as it captures the essence of the baseline rrul flent test testing both up + down and latency at the same time - and as it is written in rust, runs on everything. https://github.com/Zoxc/crusader/releases/tag/v0.0.10-testin...
dtaht··on Flent: The FLExible Network Tester
yes, had we found a way to support windows, we could have (and could still do) better. Regrettably we never did find a way - or funding - to drive tests through windows (wsl works). I still (as do many other folk in the Bufferbloat world) rely on flent to drive all kinds of tests. nothing compares to flent's ability to compare multiple network tests in so many ways.

https://blog.cerowrt.org/post/flaws_in_flent/

dtaht··on Unbloating the buffers
Ironically, we made fq-codel the default in many linuxes starting in 2012. HTB will pick up the default qdisc when you set that, so we figured many small shaping tools were just picking that up automagically.

Most people do not measure in-stream latency, rather multiple streams, so, assuming you did or did not pick up fq_codel, a packet capture will show you the tcp RTTs being managed or not. Note that on a system where the tcp is originating and then transitioning htb on the same box, another subsystem (TSQ) managed the rates fairly well before hitting HTB.

dtaht··on Unbloating the buffers
The codel ACM queue article is IMHO, better than the RFC: https://queue.acm.org/detail.cfm?id=2209336

Kathie nichols has also spoken about Codel multiple times.

dtaht··on Unbloating the buffers
Even trying L4S is rather problematic still these days. All the support for it - from the AQMs to the transports - live out of tree in an increasing ancient 5.15 release. Where cake has been shipping for nearly 10 years in some form or another. I would like that group to get on the stick on submitting it upstream, because there are some code quality problems there that they should fix (notably GRO issues), much less the algorithmic issues that I have also pointed out!

My biggest gripe about that project is that everyone focuses on the ECN change, and nobody looks at the effects on normal everyday traffic, where fq_codel or cake utterly smoke the dual-pi AQM. There actually is L4S support in fq_codel now, but they are not testing it. :(

dtaht··on Unbloating the buffers
Heh - the article refers to CAKE as a "really tortured acronym". (Common Applications Kept Enhanced). It is a backronym indeed. Some of the backstory behind CAKE is that we were shooting to compensate for some of the known flaws of fq_codel, shortly after PIE (backed by CISCO and comcast among other "big boys"), and so we called it as an internal codename, cake. To some in the project (me) it was a reference to a scene in 2010 where the russian guy keeps getting American idioms wrong: "Piece of pie" - "No, cake!". Others thought it came from the game portal, where we started saying "Bandwidth is a lie!". Anyway, the name stuck (although I wish we had come up with something more unique, google for "cake shaper" as for why), and that's where it came from.

In general fq_codel is superior to pie in just about every way. There is a specific thing that pie does that makes it appealing to hardware designers is that it is O(1) on egress and fq_codel is not (but there are ways around that). fq_codel achieves less latency than pie does, and faster than it does. codel takes a bit longer than we would like.

fq-pie still has worse tcp latency (using the same fq algorithm as in fq_codel and cake), and cake solves all kinds of edge cases that no other fq+aqm+shaper can - per host FQ (solving torrent issues), ack-filtering, nat transparency, a better codel model, easy to use link layer compensation, diffserv - and my favorite feature is actually that it runs more or less the same as a default qdisc, as a shaped one. I had hoped it would replace fq_codel at both line rate and at the shaped cases, but it did prove a bit too cpu intensive on cheesy hardware and the edge cases not noticeable enough on simple benchmarks.

I would like to multi-thread it, and add some new features, discussion here: https://docs.google.com/document/d/1tTYBPeaRdCO9AGTGQCpoiuLO...

dtaht··on Unbloating the buffers
This is a really good article. I do not know how to reach the author as there is one thing that bothers me about his default config. He set CAKE rtt to 30ms which is OK, so long as 98% or so of the sites he wants to reach are within 30ms. Otherwise the default of 100ms is scaled to be able to manage bandwidth efficiently to sites around the world, and has been shown to work pretty well to real RTTs of 280ms. In general I have been recommending the rtt be set lower with the world consisting of CDNs, but not to 30ms.

Delighted to see vyos pick it up!

Next is the shaping vs variable rate issue, on outbound and inbound. On outbound - Ideally the fq_codel or cake algorithms are located right on the bottleneck link, and respond to changes in the available bandwidth from the upstream dynamically (asserted by ethernet pause frames for example), and managed underneath by BQL (ethernet) or AQL (some wifi(ath10k,ath11k, mt76, mt79)) , though true native fq_codel support exists for the ath9k) which store up enough data only to service one interrupt. These algorithms are much (20x!) lighter weight than shaping to a fixed rate is, and like I said

Codel was designed to deal with variable rate links. A fixed shaping rate was not what we intended, but has become one of the most common and finicky use cases, dang it. It is the FQ that matters most in most cases, however.

On inbound shaping, it's a SWAG, where we recommend 85% or so of the provided rate as a starting point. You might need more, you might need less, due to the behaviors of "slow start" in particular, it is never going to be accurate enough, and we keep encouraging the ISPs merely to manage their own egress well enough (ideally dynamically and without a shaper) so inbound shaping is not needed. Libreqos/preseem/bequant/paraqum all make middleboxes that do ISP level shaping. Libreqos (for whom I work these days) has pushed CAKE to where an ISP can do 10k subscribers at 25Gbit at 50% of cpu on a cheap ($2k) Xeon or Ryzen box.

So you get something that works well on the rrul test... in the morning but not the evening. Folk then complain that their bandwidth from their provider is variable rate and shaping via their other box does not work all the time, and I go back to hoping more ISPs get told how fix it at, egress, truly right at their end.

There is another project which perhaps can be ported to vyos, called cake-autorate, which attempts to use active measurements to cope with variable rates.

dtaht··on Tell HN: Hacker News now supports IPv6
I often wish that there had been a way to politically make 240/4 and 0/8 commonly available. But in part I drove those projects to annoy the IPv6 crowd into action.
dtaht··on Tell HN: Hacker News now supports IPv6
Thank you for writing this tool. It cheers me up every time I see a green 6.
dtaht··on Queues don't fix overload (2014)
Really good thread. Several comments:

Flow Queuing allows applications not creating queues to bypass those that are. It is mildly different from fair queuing: https://ieeexplore.ieee.org/document/8469111

Load shedding, at least for packets, benefits from head drop more than tail drop. Only the codel algorithm does this but codel has been applied to other forms of queuing like Uber dealing with a concertgoer overload.

Kathie Nichols has given some great presentations lately: https://www.understandinglatency.com/

There are a lot of unbounded queues in many applications & libraries, even in rust. They make me twitchy. Even with bounded queues the number chosen is usually arbitrary. I wish a timed queue was a default rather than a length.

I highly recommend Kleinrocks work on these subjects.

I am grumpy big vendors like juniper have yet to adopt smart queues... despite the obvious benefits. https://blog.cerowrt.org/post/juniper/

dtaht··on It's the Latency, FCC
Section 706 of the Telecommunications Act of 1996 orders the FCC to “encourage the deployment on a reasonable and timely basis of advanced telecommunications capability to all Americans.” On October 25, The FCC issued a notice of inquiry (NOI) into how well we are doing and invited comments.
dtaht··on 75% of Software Engineers Faced Retaliation Last Time They Reported Wrongdoing
https://www.asce.org/career-growth/ethics/code-of-ethics is my guideline.
dtaht··on 75% of Software Engineers Faced Retaliation Last Time They Reported Wrongdoing
I try to adhere to association of civil engineer's code of ethics:

https://www.asce.org/career-growth/ethics/code-of-ethics

I wish more software folk did.

dtaht··on TCP-brutal: Congestion control algorithm that increase speed on packet lost
The bottleneck is mostly on the read path,even leveraging xdp heavily. dpdk/vpp would be faster. In the case however of an ISP 10k subscribers/network segment at 25Gbit is more than enough and busies 40% of 20 cores, on libreqos. Others are making boxes capable of more using smarter ethernet cards. If only more ISPs cared about QoE, we could cause a run on ebay on slightly obsolete Xeons and finish up fixing the bufferbloat problem right quick!
dtaht··on 25 years since the first embedded wireless Linux routers
I last wrote about the wild ride wifi took us all on back in 2010. But I had a huge flashback just now, of sitting on my roof, in the rain, in 1998,after re-aligning our antenna - and wondering where would wireless go, and how much fun we had had, in that project.
dtaht··on TCP-brutal: Congestion control algorithm that increase speed on packet lost
I think FQ helps everywhere, and FQ + aqm is needed at every fast->slow transition on the network. Ideally the core is over provisioned... except where it is not.
dtaht··on TCP-brutal: Congestion control algorithm that increase speed on packet lost
We designed cake to be implementable in hardware if anyone could be convinced it was useful: https://blog.cerowrt.org/post/juniper/ - I thought this test series rather compelling.

1) The data paths MUST prepend a timestamp to every packet 2) HW invsqrt for codel or cobalt AQM is 3k gates, checked on output 3) 32k 4 way set associative cache is a std IP block 4) HW Packet hasher - three different outputs

There are a few states in cake that are optional, but I remain boggled that step 1!! did not happen within 2 hardware releases of switch chips after van Jacobson and kathie Nichols published codel as a replacement for RED in 2012.

https://queue.acm.org/detail.cfm?id=2209336

dtaht··on TCP-brutal: Congestion control algorithm that increase speed on packet lost
Wish more would look harder there at the actual delays being observed and how bad congestion collapse gets at +250ms of it.

https://blog.cerowrt.org/post/juniper/

dtaht··on TCP-brutal: Congestion control algorithm that increase speed on packet lost
Pushing CAKE to 10Gbit/xeon core in libreqos currently (the bottleneck is on the read path actually). Or, 10k ISP subscribers per $2k xeon box.
dtaht··on TCP-brutal: Congestion control algorithm that increase speed on packet lost
Nice to see you mark! and that you are using CAKE. Cake has a feature that helps on regulating multiple flows to a single destination: per host/per flow FQ. In the middle of a network this only works well with IPv6, along the edge it has a nat mode that helps there. We put it in to manage things like bittorrent really well.

Example:

https://www.reddit.com/r/eero/comments/u7xm83/gen_2_sqm_vs_g...

The 8 way set associative hash is also much better than the direct map hash other things use, and I agree 256 flows is too small, 1024 set associative is great, 16k direct map comparable.

dtaht··on TCP-brutal: Congestion control algorithm that increase speed on packet lost
We are making significant progress with CAKE & fq_codel in the WISP & and fiber markets as a middlebox. See Preseem, Bequant, and LibreQos.io. Still best to get these native on the CPE as Mikrotik and many many others have one.

I wonder how well these would work on the GFW?

dtaht··on FTC Chair Lina Khan looks for allies and leads in Silicon Valley charm offensive
I miss bump: https://en.wikipedia.org/wiki/Bump_(application)

I rather hate the gross over-characterizations of SV as being VC-led (or destroyed). There are plenty of old fashioned network hippies still there, very concerned about the destruction of the formerly egalitarian ideals of the internet being ignored by the current ethos, people that still work on GPL'd and open source code, living in pain or poverty due to lack of credit, respect or support from the current regimes. The internet is a vast, shared resource that we hoped would benefit all personkind if we continued to understand the benefits connectivity brings to us all. I still remember and cherish the ethos that setup multiple non-profits to be its stewards - the hundreds of IXPs, in particular, but also isoc, arin and the other RiRs, places like isc.org, the internet archive, and wikipedia - all struggling to be heard over the latest grifter-driven-fad.

There is still room for entrepreneurship and innovation in SV and the world, but preferably greener and more self sustaining than it has been for the last decade.

dtaht··on 25 Years Since the Halloween Microsoft vs. Linux Memos
25 years ago today, the first of Microsoft's internal memoranda regarding how it was going to deal with the "Linux threat" were leaked to the press. Microsoft at the time was coping with a DoJ lawsuit. Linux itself, its community, and its business models, were still aborning. What of this future discussed in these leaks came to pass?

A key attack vector identified in these documents was was "De-commoditize protocols".

dtaht··on Open-source network emulators and simulators
I use sqm-scripts for a variety of purposes to insert delay or other variables like aqm and fq technologies.

I also wrote a now somewhat out of date cautionary note about all the pitfalls you can land into here: https://www.bufferbloat.net/projects/codel/wiki/Best_practic...

And over here, while we convinced systemd to adopt fq_codel by default, I cautioned against drawing conclusions from within the box simulations verses the general internet's behaviors: https://github.com/systemd/systemd/issues/9725#issuecomment-...

I am perversely glad measuring this stuff irks so many other people besides me!

dtaht··on Loadsharers: Funding the Load Bearing Internet People (2020)
Have you ever read any Buckminster Fuller? "Spaceship Earth" had a huge influence on me.
← PreviousPage 3 of 8Next →