555 karma · joined June 1, 2014
A highly interactive site like HN is inconvenient to cache with a CDN (presently). Also auth is a PITA. And HN, even at 70ms, is "close enough" to be quite enjoyable to use. On the other hand, most of my work has shifted into a zulip chat instance, which feels amazingly faster than any web forum ever could.
It would be cool if hackernews went more like a modern chat app.
Measuring packet loss is a PITA because it is the absence of information. While you can infer it from protocols like TCP, QUIC and VOIP protocols make loss only visible to the end-points.
A request response protocol like ping is a lousy metric for measuring voice quality. See irtt. Sending a VOIP-like flow in both directions, one small packet every 20ms, would be vastly better than ping, except there is an energy cost to it.
It would be great to add more to the pool.
So I would vastly prefer that in contended scenarios - an apartment building for example - that the actual channel width provisioned to be not be much more than what the ISP is providing. (and that the wifi be de-bufferbloated: https://lwn.net/Articles/705884/ )
Instead people keep provisioning 160mhz channels with dozens of other APs also living on those channels, and interference and retries go way up...
Above 50Mbits, most of the bufferbloat problem shifts to the wifi.
If more of our tests tested up + down + latency - and ISPs and users deployed bufferbloat fixes like fq_codel, cake, and libreqos.io - the internet would be a lot less jittery and feel a lot more fast, all the time.
https://blog.cerowrt.org/post/speedtests/ has some ranting. More ranting about saner uses for fiber here, also: https://blog.cerowrt.org/post/towards_better_broadband/
So the nutrition label scheme is seriously flawed without testing up+down.
So when you are talking about shaving off 6ms from a 70ms connection, it is nowhere near as dramatic as shaving off 8ms from a 10ms connection.
Now, QUIC is doing wonderful things here - 0rtt resume, and in the cloud the overall bandwidth to a given end-user is cached, but there is a really big difference between 2ms and 70! That´s just content. Geoff Huston gave a really good series of talks this week about AS paths getting a lot closer to 1 - 50 miles - across his data set, in the last 4 years: https://www.youtube.com/watch?v=gxO73fH0VqM - hopefully this will become a separate hackernews thread.
Most of my focus, however, is not on content, but communications, where we still see 300+ms glass to glass latency for videoconferencing, or worse - and bufferbloat when you are actually using the link for other things - even fiber networks tend towards 100ms worth of buffering, and fiber has a tendency to bufferbloat the wifi.... I have seen a trend in fiber towards 10ms of buffering which is actually too low and too brittle, unless fq´d and aqm´d also.
To go back to your HN example - HN loads fast because it is ONE IPv6 address (for me) and very lightweight so tcp slow start ramps up pretty darn fast, even going all the way to San Diego.
A packet capture of even a big HN discussion is really lovely, btw. More people should do that and admire it, compared to the sordid mess older websites with 100s of objects have.
too many people in his world think tcp is a function call.
ALL NETWORKS have perceptible jitter due to this, unless you rigorously apply FQ and AQM techniques to each slower hop on the path
Fatter tests like waveform show the common bufferbloat scenario in ways humans can see, better. But slow start overshoot is always there, on any connection that lasts long enough, which only takes a couple RTTs. You can clearly see netflix doing you in here, for example...
https://www.youtube.com/@trendaltoews7143
(except that in this case, the link has libreqos and FQ on it, so all that buffering is just local to the netflix flow, invisible to everything else)
However the technology exists to aim for 20ms - even 4ms - of both idle and working latency today, and the 100ms target - as well as the 95% cutoff for measuring latency, are nutty. It is not every day that key members of the ISP industry actually call for stricter metrics - If you cut the ISP working latency off at 99% instead - well, jason livinggood from comcast reported:
...
Looking at the FCC draft report, page 73, Figure 24 – I find it sort of ridiculous that the table describes things as “Low Latency Service” available or not. That is because they seem to really misunderstand the notion of working latency. The table instead seems to classify any network with idle latency <100 ms to be low latency – which as Dave and others close to bufferbloat know is silly. Lots of these networks that are in this report classified as low latency would in fact have working latencies of 100s to 1,000s of milliseconds – far from low latency.
I looked at FCC MBA platform data from the last 6 months and here are the latency under load stats, 99th percentile for a selection of ten ISPs:
ISP A 2470 ms ISP B 2296 ms ISP C 2281 ms ISP D 2203 ms ISP E 2070 ms ISP F 1716 ms ISP G 1468 ms ISP H 965 ms ISP I 909 ms ISP J 896 ms
...
As for the insane 95% FCC working latency cutoff today - in terms of real time performance - what if, you got in your car, for a drive to work, and your steering wheel failed one time in 20. How long would you live? If we want a world with AR, VR, and other highly interactive experiences, 99.9% of no more than 20ms of consistent jitter and latency should the goal for the internet moving forward, and ideally, 4ms.
A third thing, which would drive more IXP adoption, would be latency between your phone over 5G to your desktop over broadband. Everybody should be able to collect that latter metric, methinks....
And then, go measure the bufferbloat those ways.
A lot of city-based fiber dwellers think that just running fiber out to rural areas actually improves performance more than it does. Moving data and interconnectivity out there helps more.
They really should get out more. Or, perhaps, learn how to insert artificial, rural like delays into their networks in the city, to grok it.
(I am very busy today at the metro connect conference in ft lauderdale today, please expect sparse replies if any. If any of y'all are actually here today, I am outside with Bandwidth is a lie t-shirt on!
I hate the idea of wasting 6% of the original internet on rfc1918 space.
240/4 is already useful in sdns, vpns, and networks like AWS as a hop along the way. Merely moving it on the IETF/IANA ledger from reserved to unicast would recognize reality since 2008. It is hilarious how hard that has been.
Universal connectivity need not be a goal - can you imagine a portion of the internet completely immune from 30+ years of windows borne worms and viruses? :)
240/4 support in modern windows is a patch tuesday away which I would hope happen after it is moved from one side of the ledger to the other, officially. It (and 0/8) saves nanoseconds.
In doing these patches we noticed that NO IoT OS actually did the 240/4 or 0/8 check, and in deleting the checks for it in Linux, etc, 5 and 16 years ago, the internet did not melt down.
I see a lot of resentment from the router community that bemoans the extra work it will take to block these addresses, when if they don´t bother, nothing will change. Deleting code and acls for it, will actually speed things up.
The flack I took for the 0/8 patches from thousands of people and the 3 weeks it took to patch that out of everything were outweighed in terms of saving nanoseconds on every packet the first weekend that kernel deployed. 5+ years ago in the case of 0/8. Enforcing a stupid policy for decades for no reason seems silly. 0/8 should have been reclaimed in 1986.
As for what doesn´t work, who cares? Enough networks do already to make the 240/4 space usable. However AWS and google and perhaps others squatting on it!? and turning it into RFC1918 space, was certainly not my intent - I had thought that these addresses should be added to the global internet, for all mankind. Jon Postel would be spinning in his grave if he saw this abuse of this space.
I had delusions of setting up 255/8 (the most junky space available, much like 2.4ghz was for wifi) as a test, as a place to innovate, as a place to perhaps perfect udp-lite natting, and other protocols, within the open source community.
Lastly, everybody misses the most important draft of all, finally retiring the .0 broadcast idea (with no users since 1986) which will free up oodles of real world IPv4 address space. Most of that work is already done, it would take very little to finish it. Why is it so hard to get people to pay attention to that? https://www.ietf.org/id/draft-schoen-intarea-unicast-lowest-...
IPv6 also has a stupid idea about the zeroth address, only implemented in a few places that needs to be depreciated, which would shrink a lot of routing tables.
And I always wanted to visit.
but noooo.....