bcrl

Most of my career has been in software development. Running an ISP / carrier is more fun as there's more of a variety from day to day (as is the case for anything entrepreneurial) while still involving technical skills. There is a need for with some programming from time to time, but it is usually tied to solving a particular business need.

I'm sure there are other people out there frustrated with the software grind. My point is that change is always an option. There are interesting problems to solve in the world that exist outside of large software projects that most folks here have the required skill sets to tackle.

This is why I like building outside plant. You put the fibre up on poles or pull through ducts, splice it, bring it into the building, hook it up to the equipment, make sure it's working and.... you're done. It works until something breaks, usually for a very clear reason (power outage, drunk driver, rodent, vine, lawnmower man, fibre seeking backhoe, dump truck, direct lightning strike, thermal cycling of a marginal splice, failure to seal a gasket properly resulting in water intrusion that stresses fibres when the water turns into ice, ...), but those become quite rare if you're done your job properly.

On the other hand, software is never done. Even simple features, like headphones, regress these days. (I missed a meeting today because my phone decided to send audio notifications into the black void of the heat death of the universe because I didn't unlock my phone after plugging the headphones into the USB-C port of my iPhone -- the audio didn't come out of the speaker, nor out of the bluetooth of the car I was driving. No sound worked until after the phone was unlocked.)

At least with open source software I can fix the bugs I care about, but the fun goes away once you have to deal with other people to get things merged.

Is there a community of software Luddites I can go live with where we build simple technology that works and works well?

I am curious -- what is the application and the language it's written in?

There are insanely dirty hacks that you could do to start controlling the fallout of the page faults (like playing games with userfaultfd), but they're unmaintainable in the long term as they introduce a fragility that results in unexpected complexity at the worst possible times (bugs). Rewriting / refactoring is not that hard once one understands the pattern, and I've done that quite a few times. Depending on the language, there may be other options. Doing an mlock() on the memory being used could help, but then it's absolutely necessary to carefully limit how much memory is pinned by such mappings.

Having been a kernel developer for a long time makes it a lot easier to spot what will work well for applications versus what can be considered glass jaws.

It is telling that none of the online ad platforms engage with the advertising standard council type organizations that defined the standards that old school media use to self regulate. Most people don't realize that this was once a solved problem.

Implementing rate limiting in a scalable manner is harder than just making logging faster, and it makes the cloud provider money whereas rate limiting causes them to make less money. The incentive makes it pretty clear what side of the tradeoff cloud providers are going to come down on.

There's a simple solution: don't use mmap(). There's a reason that databases use O_DIRECT to read into their own in memory cache. If it was Good Enough for Oracle in the 1990s, it's probably Good Enough for you.

mmap() is one of those things that looks like it's an easy solution when you start writing an application, but that's only because you don't know the complexity time bomb of what you're undertaking.

The entire point of the various ways of performing asynchronous disk I/O using APIs like io_uring is to manage when and where blocking of tasks for I/O occurs. When you know where blocking I/O gets done, you can make it part of your main event loop.

If you don't know when or where blocking occurs (be it on I/O or mutexes or other such things), you're forced to make up for it by increasing the size of your thread pool. But larger thread pools come with a penalty: task switches are expensive! Scheduling is expensive! AVX 512 registers alone are 2KB of state per task, and if a thread hasn't run for a while, you're probably missing on your L1 and L2 caches. That's pure overhead baked into the thread pool architecture that you can entirely avoid by using an event driven architecture.

All the high performance systems I've worked on use event driven architectures -- from various network protocol implementations (protocols like BGP on JunOS, the HA functionality) to high speed (persistent and non-persistent) messaging (at Solace). It just makes everything easier when you're able to keep threads hot on locked to a single core. Bonus: when the system is at maximum load, you remain at pretty much the same number of requests per second rather than degrading as the number of threads ready to run starts increasing and wasting your CPU resources needlessly when you need them most.

It's hard to believe that the event queue architecture I first encountered on an Amiga in the late 1980s when I was just a kid is still worth knowing today.

voip.ms was pretty much offline for a couple of weeks while under a lengthy DDoS attack. They were only able to restore service by putting all their servers behind Cloudflare proxies to mitigate the ongoing DDoS.

I have cell phone calls regularly drop during tower handoffs, and codec errors that result in a blast of static upon answering a call. I can't remember a single time I had a phone call fail on the old PSTN built out of DMS10 and DMS100s locally (well, until we lost all trunks due to a fibre issue a couple of weeks ago on November 10th -- the incumbent didn't notice the outage which started at ~3:20am until ~9:30am, and it wasn't fixed until 17:38). One time when I was a teenager in the '90s, a friend and I had a 14 hour call using landlines.

The modern tech stack is disappointing in its lack of reliability. Complexity is the root of all evil.

It doesn't help that the big tech advertising platforms are making money off pushing ads peddling investment scams and malware. So long as there is a financial incentive to make money in ways that harm "users" things will only get worse. Governments have failed consumers.

Not sure how they're allowed to generate a profit or distribute dividends given the cost of the wildfires started by their complete and total failure to maintain equipment to minimum safety standards.

It's easier to write documentation than to completely rewrite a subsystem.

Circuit switching is not harder to do, it's simply less efficient. In the PSTN and ISDN world, circuits consumed bandwidth regardless of whether it was actively in use or not. There was no statistical multiplexing as a result.

Circuit switching packets means carrying metadata about the circuit rather than simply using the destination MAC or IP address to figure out routing along the way. ATM took this to an extreme with nearly 10% protocol overhead (48 bytes of payload in a 53 byte cell) and 22 bytes of wasted space in the last ATM cell for a 1500 byte ethernet packet. That inefficiency is what really hurt. Sadly the ATM legacy lives on in GPON and XGSPON -- EPON / 10GEPON are far better protocols. As a result, GPON and XGSPON require gobs of memory per port for frame reassembly (128 ONUs x 8 priorities x 9KB for jumbo frames = 9MB per port worst case), whereas EPON / 10GEPON do not.

MPLS also has certain issues that are solved by using the IPv6 next header feature which avoids having to push / pop headers (modifying the size of the packet which has implications for buffering and the associated QoS issues making the hardware more complex) in the transport network. MPLS labels made sense at the time of introduction in the early 2000s when transport network hardware was able to utilize a small table to look up the next hop of a frame instead of doing a full route lookup. The hardware constraints of those early days requiring small SRAMs have effectively gone away since modern ASICs have billions of transistors which make on chip route tables sufficient for many use-cases.

I never followed where things went after the contract was complete. Suffice it to say that we only cared about getting the protocol working, as the company was a contract engineering firm doing work for a product that was ultimately for military use. Actually testing it in the real world and improving behaviour was out of scope. We only tested it in a simulated network to make sure that the protocol correctly handled various cases (like certain wireless nodes not being able to see each other due to obstructions).

I had other friends back in the early 2000s working on WiMax, and the hardest part of their work was getting QoS right. More recently (still 10+ years ago), another friend implemented a TCP proxy for a major cell phone provider in the US that used a more wireless friendly congestion control protocol on the wireless network side of things as regular TCP breaks down when latency increases due to reception issues (which gets interpreted as congestion and triggers retransmits). Since the cellular base stations ensured that the wireless network was effectively lossless (albeit with periods of much higher latency), performance for end users increased substantially when the bulk of the TCP retransmits were suppressed.

There's a huge gap between making wireless work vs making it work well. For me, 5G is a step backwards as all the tricks used to push for higher data rates (like larger QAM constellations) make everything worse in rural areas with poor reception: there just isn't a good enough SNR 99% of the time for the new shiny, and the increased power usage does nothing other than drain my phone's battery faster than it did with older LTE. But that is where all the money for research is today.

Wireless is complicated.

Maybe people shouldn't be building data centers in deserts. In the city of Toronto the Deep Lake Water Cooling System uses water taken from Lake Ontario used for drinking water to cool a number of buildings. Most notably 151 Front Street West which houses the data centers routing most of the Internet in Ontario.

I know one company that strove for five sixes.

MANET is one of the protocols I was involved in implementing for a certain network protocol suite back around 2012. Mesh routing protocols only work for the most limited of use cases. They don't know about the capacity of the underlying wireless network and basically fall apart when things are congested or there are radios with poor reception. QoS is implemented far better in modern cell phone networks, and if the routing protocol doesn't take QoS into account, it's gonna suck.

Or we just develop it a bit more slowly. Wait 10 years and the amount of energy and data center capacity to do training runs will be substantially lower than it is today, plus the software will also achieve substantial efficiency gains. Spending too much on hardware today that will be obsolete for such purposes in 10 years seems like a mistake, and unlike the fibre built during the .com boom, there will be little residual value left in 10 year old GPUs.

Given that domain renewals can be purchased multiple years into the future, along with the fact that there are grace periods after expiration, it would take an awful lot of failure to lose a domain unintentionally. I've held my primary domain since 1997 multiple registrars and numerous hosting / colocation arrangements over the years. It sounds harder than it is if you haven't done it before.

The Archer BE3600 is uninteresting to me. It only does 2x2 MIMO which means that beamforming is not as effective as it is for 3x3 or 4x4 MIMO routers, and it therefore has worse coverage. The end result is that previous gen 3x3 or 4x4 routers are a better option for typical users. Going to 3x3 or 4x4 with 3 bands is where things become ridiculously expensive.

If all you care about is the peak speed at short range, sure, maybe it's the right choice. But for most people coverage is far more important.

The Archer C80 has 3x3 MIMO and costs under US$40. I get 400-500Mbps over wireless (which is firmly in the Good Enough territory for many uses) with it on most 2x2 devices, plus I can install twice as many for the cost of a single Wifi 7 router to get better coverage. That seems like a win to me.

The text is 1/6th of the width of my screen. It's awful. Please use more of the browser window width available whomever did this. Not everyone reads content on their phone all the time.

Any programming in real mode with compilers meant that the programmer had to annotate pointers with near, far and huge. Plus you had to know if the stack segment was the same as the data segment...... ARRRRGGGGHHH.

I am glad that time is in the distant past now, although apparently plenty of brain cells remain dedicated to the 8088.

The only person I've encountered recently that had a legitimate actual need for multi-gigabit wireless and internet access works on cleaning up LIDAR data from construction site scans. Maybe if you're editing a lot of video content, but people doing that tend to benefit more from fast local storage during the editing process.

End-user wise the only customer I've had that sustained a gigabit transfer rate for multiple days was doing something stupid: they uploaded their 20 TB NAS to a backup service, reformated the unit, then downloaded it. They could have done an in place filesystem conversion cheaper and way faster, but they chose an option in the idiot realm instead. I'm guessing they don't have backups of their data and will be very disappointed when one of the HDDs dies.

6 GHz capable access points / routers are in the Extremely Expensive realm, as a 6 GHz radio on its own is almost useless these days unless all your devices are high end and brand new. Got a security camera? No 6 GHz. Got an old laptop? Nope. What about that iPad from 2019 that still works great? Nope. Smart TV? Nope.

Very few people are using 6 GHz at this time.

I did something similar as a teenager using DOS Microsoft C to implement the Amiga's exec.library and other bits and pieces on the x86. I managed to get preemptive multitasking working in 8086 real mode, along with a serial and tty device. Later on I implemented bits and pieces of dos.library in an emulator that let me run some m68k Amiga shell binaries (like the Aztec C compiler) from a Linux command line. That was invaluable practice for learning about different execution contexts, interrupts, locking and such for Linux kernel programming. I had even written a very simple C compiler with a friend that I used to mangle C code into endian swapping code that ran natively on my 486 so that the Amiga code could call into native code to do Linux syscalls and such.

The experience of figuring these things out was tonnes of fun! There's nothing like following threads of assembly with a debugger or disassembler in the Amiga's ROM to get a better idea of how the code worked. And since systems were so much smaller in the 1980s, a single person really could understand virtually everything about the system with enough time and effort.

The biggest challenge for me was that the ROM Kernel Manuals were very expensive back then, so I wasn't able to get copies until far too late in my Amiga years (with Commodore being in its death throes).

Motorola and Intel were great back then as they would ship out printed copies of all the documentation for various CPUs and support chips for free upon request!

Good times!

Varish was very well done. It's disappointing that with HTTPS-first nowadays there is very little oppourtunity to make good use of local web caches of web content across browsers / clients. Caches would have been a godsend back in the 1990s when we had to use shared dialup to connect to the internet while using NetScape in a classroom full of computers.

MADV_FREE is also extremely expensive. CPU vendors have finally simplified TLB shootdown in recent CPUs with both AMD and Intel now having instructions to broadcast TLB flushes in hardware, which gets rid of one of the worst sources of performance degradation in threaded multicore applications (oh the pain of IPIs mixed with TLB flushing!). However, it's still very expensive to walk page tables and free pages.

Hardware reference counting of memory allocations would be very interesting. It would be shockingly simple to implement compared to many other features hardware already has to tackle.

Unfortunately, a lot of the shared state with userland became much more difficult to implement securely when the Meltdown and Spectre (and others) exploits became concerns that had to be mitigated. They makes the OS's job a heck of a lot harder.

Sometimes I feel modern technology is basically a delicately balanced house of cards that falls over when breathed upon or looked at incorrectly.

If you're bottlenecked on a 100 byte read, the app is probably doing something really stupid, like not using syscalls the way they're supposed to. Buffered I/O has existed from fairly early on in Unix history, and it exists because it is needed to deal with the mismatch between what stupid applications want to do versus the guarantees the kernel has to provide for file I/O.

The main benefit from the mmap approach is that the fast path then avoids all the code the kernel has to execute, the data structures the kernel has to touch, and everything needed to ensure the correctness of the system. In modern systems that means all kinds of synchronization and serialization of the CPU needed to deal with $randomCPUdataleakoftheweek (pipeline flushes ftw!).

However, real applications need to deal with correctness. For example, a real database is not just going to just do 100 byte reads of records. It's going to have to take measures (locks) to ensure the data isn't being written to by another thread.

Rarely is it just a sequential read of the next 100 bytes from a file.

I'm firmly in the camp that focusing on microbenchmarks like this is frequently a waste of time in the general case. You have to look at the application as a whole first. I've implemented optimizations that looked great in a microbenchmark, but showed absolutely no difference whatsoever at the application level.

Moreover, my main hatred for mmap() as a file I/O mechanism is that it moves the context switches when the data is not present in RAM from somewhere obvious (doing a read() or pread() system call) to somewhere implicit (reading 100 bytes from memory that happens to be mmap()ed and was passed as a pointer to a function written by some other poor unknowing programmer). Additionally, read ahead performance for mmap()s when bringing data into RAM is quite a bit slower than on read()s in large part because it means that the application is not providing a hint (the size argument to the read() syscall) to the kernel for how much data to bring in (and if everything is sequential as you claim, your code really should know that ahead of time).

So, sure, your 100 byte read in the ideal case when everything is cached is faster, but warming up the cache is now significantly slower. Is shifting costs that way always the right thing to do? Rarely in my experience.

And if you don't think about it (as there's no obvious pread() syscall anymore), those microseconds and sometimes milliseconds to fault in the page for that 100 byte read will hurt you. It impacts your main event loop, the size of your pool of processes / threads, etc. The programmer needs to think about these things, and the article mentioned none of this. This makes me think that the author is actually quite naive and merely proud in thinking that he discovered the magic Go Faster button without having been burned by the downsides that arise in the Real World from possible overuse of mmap().

If you're doing large amounts of sequential reads from a filesystem, it's probably not in cache. You only get latency that low if you're doing nothing else that stresses the memory subsystem, which is rather unlikely. Real applications have overhead, which is why microbenchmarks like this are useless. Microbenchmarks are not the best first order estimate for programmers to think of.

None of those APIs are cheap enough to call in a fast path.