HNHacker News
TopNewBestAskShowJobs

pmjordan

7,982 karma · joined January 21, 2008

British-Austrian software developer, happiest when on a travel adventure.

While looking for the next big thing, I do contract/consulting work; I prefer lower-level stuff, such as device drivers/kernel hacking, real-time graphics and game tech.).

You're welcome to email me about anything, and I promise to reply if you stick to one topic or question and it isn't too open ended. Specifically:

I get a lot of email with technical questions that would require me to download and read the documentation for libraries X, Y and Z to answer - please don't do that; instead, put the code that doesn't work (or pseudo-code that you'd like to turn into real code) in the email, and quote the prototypes for any used APIs, ideally deep-linking to their documentation just in case I do need to look at it.

Obviously, if you're planning to pay me for my help and advice as a consultant you may ignore the above. :-)

website: http://philjordan.eu/

email: phil+hn at philjordan dot eu

fedi/mastodon: @pmdj@mstdn.social https://mstdn.social/@pmdj

github: https://github.com/pmj

submissionscomments
pmjordan··on A months-old AMD microcode bug destroyed my weekend
I believe Wireguard is implemented in the kernel. I'd argue there should be a kernel-wide wrapper function for this sort of thing, or if that already exists, Wireguard should probably use it.

That doesn't explain why systemd uses it, of course.

pmjordan··on Apple Developer Documentation Is Missing
They have credited a bunch of DTS incidents back, but that normally only applies where they determine that they either:

* can't help you

* don't want to help you

TBH, compared to the person-hours required to write up the DTS incident in the first place and keep the conversation going, test out their responses, etc., the cost of the incident itself is minimal.

Let alone the cost of the times where I decide to go it alone and figure it out by trial & error or reverse engineering…

pmjordan··on Text Editing Hates You Too
I assume this is about having to use other people’s devices, which aren’t configured how you’d configure your own.

I personally have muscle-memory for my own programming-optimised adaptation of Type 2 Dvorak, regular German layout, and UK/US/International variants of English keyboards, each in Mac and PC variants. And, I suppose, in mobile phone and tablet screen-keyboard variants.

It always takes a few seconds to adapt to whatever I’m sitting in front of, and I have a lot of practice with this compared to the average person.

At least in German, there are standard substitutions for when umlauts and sharp S aren’t available for technical reasons:

ä -> ae

ö -> oe

ü -> ue

ß -> ss (historically “sz“, which you still occasionally see, particularly where there is Hungarian influence; Swiss German tends to forego ß altogether)

pmjordan··on Apple Developer Documentation Is Missing
Driver development docs on macOS largely haven't been updated in 10+ years. I started doing macOS kernel stuff when Snow Leopard was the latest release, and the only documentation I can think of since then has been sample code for Audio Server Plugins (for writing new-style audio drivers) and headerdocs explaining the transition from IOUSB* to IOUSBHost* APIs. Any additional information has been buried in a bunch of WWDC session videos, or you have to extract it from Apple by filing DTS incidents.

On the other hand, Apple have decided a bunch of documentation was out of date, so they've simply made it harder to access by burying it in archive sections or taking it offline altogether, without providing replacements. Getting into driver development now is almost certainly a lot trickier than it was back then.

pmjordan··on Ask HN: What are unintended consequences of new tech you've noticed?
For Austria at least, their car travel estimates are eerily accurate in my experience, including accounting for normal levels of traffic for that time of day/week. Traffic jams mess with it to some extent of course, as they appear and disperse relatively suddenly. And no, I don’t speed.

Their bike estimates are consistently wildly optimistic however. If it’s an easy route and I pedal flat out with no traffic and without hitting any red lights, I can get close to the estimated time.

OK, I don’t have a road racing bike or e-bike, and there are people who are more athletic than I am, but I’d say I’m fitter than the median. Their model seems to assume around 25km/h average speed, which isn’t really sustainable (Or safe) in everyday cycling with other road users about.

pmjordan··on Why iOS 13 and Catalina Are So Buggy
I generally call the user-space portion of a driver that's loaded into applications to be plugins and keep them outside of the umbrella of "drivers" but in this case I think we both understand what's going on and are just disagreeing on what the terminology is.

Given that the kext is essentially useless without the userspace component (and vice versa), I don't think it's unreasonable to call the plugin part of the driver. Note also that it's installed in /Library/Extensions. This certainly also matches the user experience: they download the NVIDIA "driver" installer which happens to install a bunch of kexts and also the user space components at the same time.

These are plugins that implement AudioHardwarePlugIn, right?

Correct.

More generally, the IOCFPlugin mechanism, which paired user space components with kexts, is essentially dead for third-party driver developers thanks to the increasing lock-down of user space processes.

pmjordan··on Why iOS 13 and Catalina Are So Buggy
Library validation has nothing to do with drivers.

Depends on the type of driver. For GPU drivers, it is indeed a problem.

I'm going to assume from your comment that you haven't actually written many (any?) device drivers on macOS, so let me explain roughly how GPU drivers on macOS work:

- One or more kexts provide an IOFramebuffer implementation, plus an IOAccelerator object which listens for various types of user clients. The IOFB and IOAccel are typically clients of the GPU's IOPCIDevice object and handle its interrupts and memory.

- A variety of user space plugins (essentially dynamic libraries) are loaded into any process wishing to use GPU acceleration, with different plugins for Metal, OpenGL, and so on.

The kext communicates directly with the hardware and manages memory, but the user space plugin tends to deal with the higher level tasks such as generating command streams, compiling shaders, etc., and they talk to the kernel driver using the IOKit user client mechanism. The plugins also provide the back-end implementations for the app-facing APIs such as OpenGL, Metal, and OpenCL.

In recent macOS versions, WindowServer, the process responsible for drawing your macOS desktop, has had the library-validation codesigning flag enabled. This flag enforces that any code loaded into the process must be signed using a certificate from the same Developer Team ID as the main executable or Apple's own certificate. This means, no third-party plugins can be loaded.

This also means third-party GPU drivers can't work with most Apple apps and WindowServer itself, because they require aforementioned user space plugin.

For similar reasons, classic Core Audio plugins (not to be confused with Core Audio Server Plugins) are deprecated - they used to be loaded directly into the process using Core Audio, which no longer works on apps with library-validation enabled.

Drivers which are implemented entirely as kexts or which are loaded into a "server" user space process are not affected by this, which I guess is what you were talking about.

pmjordan··on Why iOS 13 and Catalina Are So Buggy
It would be very nice to be able to use libvirt from my Mac in the same way as my Linux workstation.

I believe libvirt as such is supposed to work on macOS. Furthermore, recent versions of Qemu apparently support using macOS's Hypervisor.framework. So it seems like this should be doable, although I've not had a chance to try it myself yet.

pmjordan··on Why iOS 13 and Catalina Are So Buggy
The macOS kernel (XNU) does not have any kind of containerisation support. I believe "Docker for Mac" is actually a Linux kernel running inside a hyperkit/xhyve VM, and uses Linux's container support.

Apple hasn't put much public effort into making macOS work well inside a virtual machine. Recent versions in particular, where the window compositor uses Metal for drawing, don't run well. Apple hasn't opened the API for writing GPU drivers, so there are no drivers for the virtualised GPUs provided by VMWare, Parallels, VirtualBox, or Qemu in their VMs. (The tightening of code signing requirements [see the 'library-validation' flag] even locked out nvidia from providing their own GPU drivers from macOS 10.14 onwards.) The only way to have GPU acceleration in macOS guest VM is to pass through a whole GPU at the PCIe device level. (I don't know if anyone has tried AMD's MXGPU tech, which allows GPU 'slices' to be passed to VMs, with Macs and macOS VMs; the GPUs that support it are a little too expensive for a weekend project.)

There are signs that Apple does use virtualisation internally; specifically, they've started adding drivers for virtio devices and one of Qemu's display adapters to macOS.

pmjordan··on Estonia is running its country like a tech company
those are all matters of how those transit agencies are run, not issues with the payment system.

I think that was the point of the originator of the thread - the technology is there, it's just not applied properly. It may have improved recently, but when visiting SF for a few weeks in 2013 and 2014 I was left scratching my head about how best to charge my Clipper card for Muni transit. Topping up the credit via the website had a waiting time of 1-2 working days, so if you were trying to catch a tram or bus into the city centre from the outer neighbourhoods, you had to plan ahead for having enough credit, especially when the weekend was coming up. At BART stations it was usually less problematic to charge the cards - assuming the machine worked, which wasn't always the case.

At any rate, the whole thing felt very uncivilised and anachronistic compared to my experiences in much of Europe, Japan, Hong Kong, and Taiwan. (Japan's Pasmo and similar cards require cash for charging and largely don't accept credit cards, but you'd be hard pressed to find anywhere where you'd have to walk more than a block for an opportunity to charge them - or to find an ATM for obtaining the cash you need.)

pmjordan··on Estonia is running its country like a tech company
Yes, the country/countries where your physical office(s) is/are located (or rather, where people - founders or employees - are doing the work that is the basis of the revenue on which you're generating a profit) will likely want a piece of the pie.

Certainly, a decade or so ago, it was popular to register a limited company in the UK whose physical presence was on the continent. Invariably, this meant dealing with both tax authorities.

It seems to have lost its attractiveness after laws for GmbH and similar were relaxed in various countries. Brexit has pretty much killed any remaining demand, at least for real companies. (As opposed to shell companies for tax evasion or money laundering purposes, for which the UK remains as popular as ever - and which are a major reason for big money backing Brexit, as the EU has been looking to crack down on the practice.)

pmjordan··on The Decline of Stack Overflow
I don't know about employment, but Stack Overflow has become one of the main lead gens for our 2-person software development consultancy. I monitor the tags that are relevant to our niche and try to answer any sensible questions that come along. (And try to help out people with less sensible questions where I can too.)

This presumably only works if you you cover some obscure-ish, deeply technical niches, not for more run-of-the-mill stuff. If your clients are non-technical, they won't find you on Stack Overflow. If your clients are companies that develop software themselves but need specialist skills for a project, it can work.

Note that I don't think my actual SO score[1] is in any way of importance. I suspect it's more important that:

- The answers are actually well written, in-depth and helpful.

- I have a reasonably recogniseable avatar image. I have been told by eventual clients that they kept running into my SO answers while they were trying to solve their problem themselves. Presumably this pattern is much more noticeable if there is a human face staring back at them from the bottom of the answer.

- Friendly comment exchanges probably help. Being an arse in SO comments probably won't help your chances of being hired by the person who asked the question or had the same problem and found your answer via a web search.

[1] At around 14000, my reputation score puts me in the top 3%, but if the stats in the article are correct, that doesn't even put me in the top third of the 8% of many-question-answerers. (I have answered 518 questions.) Mind you, my mean answer score is probably about 2, with the median squarely at 1; niche topics mean few people read my answer, and even fewer upvote them. My only gold badge is for voting on 600 questions, but I have bronze badges coming out of my ears.

pmjordan··on Apple of 2019 is the Linux of 2000
Backups (both hardware and data) or not, a month turnaround is still awful and unacceptable, and reason enough to switch suppliers. Companies that take business customers seriously offer same-day on-site service.
pmjordan··on The Isle of Man is not in the UK
while I had obeyed all the rules at the time, they were retroactively changing the rules and I needed to declare all the interest I had earned on my offshore account over the last 8 years.

Are you sure they changed the rules? It sounds like they might have realised they hadn't been implementing/enforcing the European Union Savings Directive (EUSD) [1] correctly, and therefore retroactively enforced the rules. Or perhaps they found that the EU rules overrode the UK/IM rules, which were retroactively found to be null and void.

[1] https://en.wikipedia.org/wiki/Savings_Directive

pmjordan··on Prehistoric Babies Drank Animal Milk from Bottles
I thought babies produced more lactase (lactose-digesting enzyme) anyway, even for people (and ethnic groups) with lactose intolerance in adulthood? Human breastmilk contains lactose - so the body expects to have to break it down at that age.

(I'd have thought any preference for goat's milk over cow's would be connected to the protein it contains.)

pmjordan··on Four common mistakes in audio development (2016)
I've seen variable size accesses (sometimes 32-bit, sometimes bytewise) to memory locations (almost certainly field access via a dereferenced struct pointer, i.e. operator ->) when reverse engineering x86-64 code known to have been compiled with clang. However, that was very specifically for bitwise access - I don't know if the original source code used C's bitfield struct syntax or explicit masking. I also don't recall if it was just reads whose access size was inconsistent or writes too. It's almost certainly a code size optimisation when comparing to or storing immediate (compile-time constant) values.

Either way, if targeting a modern toolchain, I would strongly recommend using atomic_store()/atomic_load() from <stdatomic.h> for exchanging data between threads when "stronger" interlocked operations (CAS, exchange, atomic arithmetic, etc.) aren't required.

Be very, very careful with the memory order argument when using the "explicit" versions of these functions. Apple subtly broke their IOSharedDataQueue (lock-free & wait-free circular buffer queue between kernel and user space, used for HID events, for example) in macOS 10.13.6 (fixed in 10.14.2 I think) because they used memory_order_relaxed for accessing the head and tail pointers when transitioning to the stdatomic API from OSAtomic. Presumably in a misguided attempt at "optimisation". The writer would only wake up the reader if the head/tail state before or after writing the message to the buffer indicated an empty queue. Unfortunately, due to memory access reordering in the CPU, the writer ended up sometimes seeing a state of memory that never existed on the reader thread, so the reader would wait indefinitely for the writer to wake it up after draining the queue.

The use of volatile for exchanging data between threads, as recommended elsewhere on this thread, is iffy - make sure you know exactly what volatile means to the compiler you are using and the version of the C/C++ standard you are targeting.

pmjordan··on Colorado Town Offers 1 Gbps for $60 After Years of Battling Comcast
The EU is a big(ish) and diverse place. We pay about twice that for 40Mbps VDSL and about 2.5x that for 50Mbps LTE. To be fair, we could get something slightly (~20%) cheaper as we've got a "business" deal to get static IP addresses on each connection. The fastest consumer package we could get at this location (cable) is 500Mbit/s down, 40Mbit/s up for €80/month. (Their business level contracts top out at 300/40 Mbps and €200/mo)
pmjordan··on We Have Ruined Childhood
Having close connections to both countries, I can say that kids in Austria are definitely freer than in the UK.

For one, in Austria, kids have a legal right to go to school on their own, be it on foot or by public transport. A lot of parents do drive their kids to school, unfortunately even where it's really not necessary, but you do also see plenty of young kids walking there or home, or taking the bus or train. You definitely also see more kids "hanging out" in Austria - in the UK, everyone seems quick to jump to conclusions about gangs, so I guess "respectable" parents won't let their kids do that.

On the other hand, I felt I probably had even more freedom growing up in the 90s (in Austria) than many of today's kids. This is probably more of an individual choice of the parents (for some reason, people are afraid, despite this being one of the safest countries in the world) than direct societal pressure (being reported to the cops or social services, etc.) as it seems to be in the UK or US.

pmjordan··on Germans Grow Frustrated with Their Slow-Poke Internet
Perhaps you'd like to explain how people driving through the country, or anyone else in fact, drives down ADSL speeds to single-digit or low tens of Mbit/s. Or indeed how population fluctuations of about 1-2% might cause cellular networks to develop dark patches or to regress from LTE to 3G or EDGE.
pmjordan··on Toilets of the World
As a Central/Northern European I noticed the same thing when spending a few months in North America. Particularly the "low-flow" toilets in the US & Canada which seem to be considered an improvement(?) have a ludicrously small outflow.
pmjordan··on Ancient History of Lyme Disease Revealed with Bacterial Genomes (2017)
Thanks!

My email address is in my profile. In case she wants to get in touch, feel free to pass it on, I'm happy to answer any questions etc.

(This also goes for anyone else reading this who has or suspects they might have Lyme disease, or who has someone close in that position; I'm obviously not a doctor but will do my best to help.)

pmjordan··on Ancient History of Lyme Disease Revealed with Bacterial Genomes (2017)
but if your tests for Lyme were negative, it wasn't Lyme (borreliosis).

Depends on the test. The ELISA test is cheap but extremely unreliable. ELISPOT/LTT seems to be the most reliable antibody test out there, at least for European strains. If the latter also came back negative, the evidence would indeed suggest this wasn't a Lyme infection but something else.

pmjordan··on Ancient History of Lyme Disease Revealed with Bacterial Genomes (2017)
Fellow chronic Lyme patient here.

I was given 2 rounds of doxycycline which seemed to do absolutely nothing.

1. Doxy is not the worst antibiotic for targeting lyme that's got into the central nervous system, but from personal experience and the doctors that have treated me, it seems Minocycline is preferable. Note that it may seem like you're having a bad reaction to the medication, but this is probably just the Jarisch–Herxheimer reaction. I was told to gradually increase the daily dose to make it more bearable.

2. Once the infection gets to the chronic stages, a single type of antibiotic will at best suppress the infection, it will normally not get you anywhere near a cure. Different doctors will have different go-to combos, but should generally be flexible about it and try different combos if one isn't really working. Again, this should also be influenced by the affected organs based on symptoms. I'm glad you're now getting such a combo, many doctors who don't specialise in Lyme disease will not prescribe that kind of thing even if there's now a decent body of literature that this does work in many cases.

I've been on 3 antibiotics (simultaneously) for the last 2 months and I feel so much better - not 100%, I'd say 85%.

I've been there - be prepared for this potentially being a long slog, depending on severity and affected organs. After first developing symptoms in 2013, I finally got a diagnosis in early 2015. (Backed up by positive test results - Lyme ELISPOT/LTT, the ELISA test shows nothing, apparently it's only about 40% accurate) At that stage I was pretty bad, I have almost no memories from late 2014. The doctor that first diagnosed me got me to a stage where I was symptom-free whenever I was taking antibiotics, but after 2-12 weeks symptoms would return.

I ended up switching doctor in late 2016 as my overall progress seemed to have stagnated. She turned out to be a lot more test-happy than the original guy, and tested for a whole range of other stuff - bacteria, fungal, parasite, viral infections and various other values related to inflammation, nutrition etc. She found that in addition to Lyme disease (2 different strains of Borrelia), my immune system was also in overdrive fighting a bunch of other stuff, which I got treatment for. Plus she switched me to a different antibiotic regime specifically targeting one of my main symptom areas (bladder/prostate). 2 1/2 years later I'm not 100% cured. One of the 2 Borrelia strains I was infected with seems to be gone. The other one is still lingering - I've just relapsed into a mild Lyme episode for the first time after 6 months of not taking anything, so I'm on another 3-antibiotic treatment cycle. The gaps are getting longer, and the symptoms are becoming less severe. My immune system still occasionally goes wild on fighting VZV too (chicken pox virus, once you've had chicken pox it stays in the nervous system) which presumably somehow gets an opportunity to flare up thanks to the Lyme bacteria doing their thing or my immune system generally being in a rough shape after years of infection.

One thing that seems to have helped me is doing regular intense exercise; I've taken up tennis, but anything that gets you properly sweating should do. The recommendation was actually to do regular infrared treatment, but I hated that, and intense exercise seems to work similarly well. The perspiration supposedly helps to get rid of the toxins from the bacteria (live or dying, see Jarisch-Herxheimer reaction) and the spirochete bacteria themselves apparently cope badly with a raised host body temperature. (An early treatment for syphilis, another spirochete, was apparently the high fever produced by contracting malaria. I don't recommend infecting yourself with malaria.) I have no idea how effective it is at those 2 claimed benefits, but it certainly seems to clear the brain fog for 2 days or so.

FWIW, no dietary modifications seem to have made any difference for me; the first doctor made me try a bunch of these but the change after a few months was just that I felt hungry a lot.

pmjordan··on Ancient History of Lyme Disease Revealed with Bacterial Genomes (2017)
This may or may not be obvious, but the Lyme vaccine for dogs is not the same as the discontinued one for humans. So it's (unfortunately) not a simple case of rebranding/repackaging the vaccine they're already producing for human use.
pmjordan··on To plot my next murder, I went to the Body Farm
Fortunately she had quite the grim sense of humour to help her deal with the cases she worked on.

Unfortunately, even a sense of humour doesn't protect you from PTSD when dealing with this sort of stuff, particularly on a daily basis. I recently listened to the audio book version of Dr Richard Shepherd's "Unnatural Causes" [1]. It's all very fascinating, and he's clearly passionate about his work while also providing a great service to society as a whole but as becomes clear, it has also affected his mental health and family life.

[1] https://amzn.to/2LBaOF4 - (The audio book is excellently narrated by the author himself; that doesn't always work out well, but it does in this case.)

pmjordan··on Jony Ive to form independent design company with Apple as client
Maybe it's a technological limitation?

I've heard this second-hand (or third-hand) and my EE knowledge is nowhere near good enough to say how accurate this is, but: Apparently the power delivery negotiation mechanism in USB-C is vastly more complex than for USB-A, so hubs with lots of USB-C ports would need new, more complex chips compared to equivalent USB-A hubs. It already took a few years for reliable hubs to turn up when we first got USB 3.0. I'm assuming the manufacturers might also be trying to hit USB 3.1 Gen2 speeds at the same time as supporting USB-C, which presumably further tightens reliability tolerances. I'm guessing with all that, they might not be able to hit the desired price points with this stuff yet, and all that complexity probably also takes a lot of power so bus powered USB3.1 hubs with USB-C downstream ports are presumably a tricky proposition too.

pmjordan··on VESA Publishes DisplayPort 2.0 Video Standard
Once you've started this process, I believe you can just move the mouse (without clicking) to move the window as well. This tends to be faster than using the arrow keys, especially at modern display resolutions.
pmjordan··on 7nm AMD EPYC “Rome” CPU with 64C/128T to Cost $8K (56 Core Intel Xeon: $25K-50K)
As a driver developer, I've been on the receiving end of stuff breaking due to changes in macOS many a time. Based on my experience, I find it much more likely that Apple simply does not care about this problem/does not consider it a problem. By that I mean they are not willing to assign developer resources to the situation because from their point of view it is not a supported configuration, plain and simple. The fact that it ever worked is largely accidental as far as they are concerned.

One of their highest priorities at the moment appears to be to remove any 3rd party code running in the context of Apple processes. For examples, you don't need to look far: kext -> DriverKit/System Extension transition, hardened runtime, etc. The Nvidia driver breaking is just a consequence of that increasing lockdown. The simplest solution would be to allow nvidia's code to run in the context of core system and user processes again (kernel, WindowServer, just about any GUI app) which presumably is not palatable to them or they wouldn't have made the change in the first place. The alternative would be to farm this type of driver out into a sandboxed user process. (Nowadays this means DriverKit.) Aside from potential performance issues, this would be a lot of work, so it's probably never going to happen unless literally millions of end users start complaining.

pmjordan··on 7nm AMD EPYC “Rome” CPU with 64C/128T to Cost $8K (56 Core Intel Xeon: $25K-50K)
Correct, the built-in drivers only cover the "Kepler" architecture, as that was the last one that Apple used in a Mac. The 3rd party Nvidia drivers which support more modern GPUs are only offered up to macOS 10.13 High Sierra.

The technical background for this is that Metal and OpenGL acceleration requires the GPU driver bundle to be loaded into a process's address space. (effectively a dynamic library) With 10.14, Apple has enabled the "library-validation" codesigning flag just about everywhere, which means that only libraries signed by the same developer as the process's main executable or by Apple can be loaded into a process. Hence, 3rd party GPU drivers won't load in WindowServer for example, which makes them dead in the water.

Apple could of course implement an exception or sign Nvidia's driver, but so far it looks like they're not interested. Perhaps there will be a solution to coincide with release of the new Mac Pro which once again contains PCIe slots you conceivably might want to fill with Nvidia cards, but I wouldn't count on it.

(FWIW, my company maintains the macOS driver for one of 2(?) manufacturers of USB graphics adapter chips, so I have a pretty good idea of how the macOS graphics stack works.)

pmjordan··on AMD Ryzen 5 3600 Beats Intel’s Core i7-9700K in Cinebench
I think it might have worked in some earlier generations of Core i, perhaps Sandy Bridge and Ivy Bridge.
← PreviousPage 3 of 34Next →