Docker Desktop Broken on Mac OS Update for over a Week | Hacker News Reader

Docker Desktop Broken on Mac OS Update for over a Week

dockerstatus.com

66 pointsJohnMakin75 comments

Ran into this today. Took me about an hour to figure out. You have to run the script they provide (can’t find the link right now but it’s in the GH issue). But if you are like me and installed it via homebrew, you also you have to uninstall docker fully and you have to do it via homebrew. So instead of upgrading you have to `brew uninstall docker` and THEN brew update & `brew install —cask docker`. Bit of a pain but it does eventually sort itself out.

looks to be this: https://github.com/docker/for-mac/issues/7527#issuecomment-2...

Installing 4.37.2 does sort the malware warning, for those who install with the binary. Was a fun post holiday treat for those who have lots of developers.

yes. it’s been a stupid amount of work for something I could not possibly care less about, but the issues for me are not the fresh installs of the patched version, but uninstalling corrupted versions.

Judging from the script linked here (https://github.com/docker/for-mac/issues/7527), Docker’s Developer ID cert was revoked by Apple. Are…they going to explain why/how that happened?

It's also possible someone with access to the Docker team account on the Apple developer portal revoked it by accident.

The access controls on the Apple developer portal are not particularly fine grained, there is no audit trail, and many actions cannot be undone.

That’s what I thought at first, but actually this seems to be not the case for Developer ID certificates? https://developer.apple.com/help/account/reference/revoking-...

Correct. Developer ID certs cannot be revoked self-service, you have to email Apple.

Gets out soapbox

App Store Developer portal is a mess. I manage 10+ accounts and I hate using it so much (Google Play isn’t better, just sucks in different ways). I also have access to an enterprise account and “nerve racking” does not even begin to describe what interacting with the portal is like.

A single click with a badly worded confirmation step is the ONLY thing keeping your released app in the field from shutting down until you get a new version out. Let me say that again: if you delete your signing cert then your app will not open on devices anymore. No way to communicate to users to update or let them know anything at all. Just dead.

Even if you avoid making that mistake, the cert rotation dance is horrible and terrifying, it doesn’t help that Apple’s docs and help text are completely useless. It doesn’t help that most info you find online is for AppStore certs/signing and not Enterprise certs/signing (the _massive_ difference being you can revoke a cert for an App Store app and it keeps working). There is no wording about max certs on their webpage so in most cases you just have to try something in the portal and see if it fails. It’s even more fun because every click in the cert rotation process is a gamble and so after you’ve worked yourself up to finally clicking the button you then run right into an error and you have to rethink your strategy. Again, your entire business potentially hangs in the balance if you pick wrong, have fun!

I understand what Apple is trying to accomplish and that’s fine but they need better messaging. They need to make it clear what the effects of revoking/deleting certs/profiles is. They need to make the connections between certs and profiles more obvious.

The whole UI is horrible and flakey on top of that.

I wanted to leave the original post unsullied by my ragy commentary:

They are listing this as "operational" but the steps they are outlining here are not a proper solution for this problem! I cannot sudo on my work macs. How can a severe problem like this go on for more than a week? This has completely shut down my local development. I'm mad at the situation more than anything, but also a big middle finger to mac OS for not allowing me, a somewhat mature professional adult with expertise in stuff like this, what I want on my machine. My new workstation orders are not going to be macs. Good job!

The worst part is, even if you go with the extremely user-hostile un-minimizable prompt wants you to do, which is delete a bunch of files, the popup will still spam you until you are able to follow the instructions on the remediation that docker provided! Completely FUBAR

John, it's really pretty easy to fix:

Take a deep breath.

Uninstall docker, including from homebrew if you've ever used that to install it.

Reboot

Download newest ARM version from Docker.

Install it.

Reboot again.

You should be good.

If it wasn’t clear in my original post, I have to solve this for a fleet of machines. I’ve taken many breaths today, but thanks for your condescending concern. And as I said, for people with typical machine access in an org, this is a completely unacceptable solution. Anything that requires sudo access and a pasted bash script in a doc is a completely unacceptable solution. You cannot turn around and call this solved.

What you can do is have a cron job that continuously polls one or more URLs looking for tarballs with Git repos accompanied by GnuPG or Signify signatures. The primary idea is that the tarballs are only unpacked and executed if the signatures are valid and anything received by the machines is only executed after the hashes and signatures are successfully logged, this prevents (or at least documents) any abuse on the part of the IT team. Inside the tarballs are Salt, Ansible, and/or shell commands.

In addition to a cron job, we had an active pubsub listener on each machine to poke this process.

You can do this on macOS too, or at least, modern macOS "should" be able to do this.

We did this on a large fleet of user machines (laptops) running Windows 10/11, macOS, and Linux.

On an entirely separate tangent... Why are you remotely managing macs? That seems like a cluster of nightmarish proportions without even getting to this issue.

If you don't trust people, give them Windows and use the tools that have been around for decades for this.

Remote management is a requirement for SOC other compliance frameworks.

That doesn’t sound right. Most of SOC compliance boils down to having a plan to check enterprisey checkboxes.

So, you pay someone that can write a report explaining how the boxes will be checked in a way that minimizes the blast radius of security theater stuff like broken approaches to remote management. Consider fixing your soc compliance plan.

Anyway, assuming the remote management tool has root, you can use brew to install docker desktop from the cli.

In principle yes. The implementation however is rarely defined (if you have examples, feel free to share). In my experience it's by choice that companies use compliance to turn their dev environments into a micromanaged hell that engineers hate working in (our company went through multiple compliance certifications including SOC1 and SOC2).

Not at the level John was talking about. My last company had SOC 2 and yes, there was remote management, but the developers still had admin access to their machines, and didn't need some BOFH to use docker.

I agree that requiring sudo access may not work for some orgs at a user level. But it also seems unusual that you need to solve this for a fleet of machines and yet are not allowed sudo access.

I dont know how it could be less obvious that I manage other users’ devices, and yes, I can escalate my permissions fine, but rolling this out to a distributed fleet of machines, some of which may not be remotely in the same timezone, is not quite as easy - there are many situations where sudo is not granted by default to a workstation. I’m amazed people find this surprising in this thread.

If you're managing Macs on anything like an official basis, why are you not using any of the various tools that exist to enable and streamline that?

Macs have built-in device management tools, and there are third-party solutions like JAMF that can definitely run scripts like what's needed to fix Docker, as root, including reboots if necessary.

It really sounds like the primary problem you're experiencing is not what Macs allow, nor even the shoddiness of the Docker solution (which I will not attempt to defend, I agree they should have something more robust), but your company's policies that put you in a position of responsibility but no power.

sometimes you deal with contractors that you don’t have such explicit and unrestricted access to their devices. but yes, being in a position of responsibility and no power is basically the IT sysadmin’s creed, and that is not even my official role, just one I am forced to play a lot.

And this is a problem with company policies, not with the design of macOS or Docker.

I'm sorry, but there's just no sane system design that can make up for such abjectly counterproductive policies, no matter how common they may be.

If you are in the org with managed macs, you should not be doing this yourself, your IT (or security, or whomever manages your computer) should be creating a script and adding it to whatever secure script runner they deployed to user machines. Or even better, automatically fixing this.

If you are in the org where your IT cannot do this, and the same IT took away your "sudo" access, then there is nothing you can do. Even if there were a hacky way, it would be against policies, so do it by the book and wait for IT to fix it for you. If you are on important and urgent project, escalate via official channels. If you are not on important project, relax and don't forget to write "I could not do anything because I am blocked on IT" in all of your status reports.

If the previous paragraph makes your blood boil or makes you want to cry, consider working for a different company which either gives autonomy to developers or has better IT.

I have literally switched to a lower paying job because of IT issues at a previous firm. The extra headaches was not worth it to me. It doesn't help IT was full of Microsoft fanboys over there who saw Macs as a nuisance and did barely any testing on them of their updates

> should be creating a script and adding it to whatever secure script runner they deployed to user machines. Or even better, automatically fixing this.

It may surprise you to learn that a lot of orgs have critically understaffed or unknowledgeable IT teams. Yea, I could "solve" world hunger by simply feeding everyone, but how do I actually do that?

I am the crititcally understaffed "IT" that has to fix it for you. And now it's urgent because local development has basically halted on tons of devices for a variety of weird and undocumented reasons. I am trying to communicate to you that rolling out a "fix" for this type of thing to a distributed fleet is not trivial at all. For a typical IT sysadmin this is one of the most annoying possible things, at the worst possible time too (post holiday), and that's not even my official role! I just have to play it a lot. Also, I think you're severely underestimating how complicated it is managing a remote fleet of devices under current compliance and security standards across the industry. You're talking from the perspective of a dev that gets mad they can't have unlimited access to the prod db with a single ticket, I think, but the reality is these things are not easy to roll out on the fly under specific circumstances. Good for you that you found a company that fits your needs/whims and isnt bound by these same kinds of restrictions. I'm also not making any commentary on whether any of this is sane at all, just that I am the guy at the short end of the stick that has to deal with it and I'm (justifiably) pissed off about it.

Judging from the fact that all the workaround does is reinstall a privileged daemon, you needed root to install Docker in the first place. Why are you outraged that you need root again to reinstall? Moreover, how is that even a macOS problem?

Being outraged that you need to invoke administrative privileges on a development machine seems unreasonable to me. You can't get far with that kind of restriction.

because modern macOs does not gracefully allow true root, without hacking, in a way that doesnt solve compliance requirements?

macOS obviously has the capability to reinstall Docker. If your company is preventing you from upgrading to the fixed version in timely manner, that's your company's responsibility.

Perhaps you should inform yourself about the actual problem/proposed solution here - the issue isnt installing a fresh new version, it’s the uninstallation steps required to even get there in the first place. and no, mac os will not let me do this, even when I follow the prompts. docker’s elevated permission scripted solution is the only thing that has worked yet.

Looking at the incident page, the bash solution was superseded by a proper release the next day. And the linked document specifically instructs to "update directly through the app" if possible [1]. So uninstall not working is completely news for most people reading this thread. Even the title of this thread, "Docker Desktop Broken on Mac OS Update for over a Week," isn't substantiated anywhere else.

But even if it is required to manually stop and remove the two daemons, shouldn't that be doable using any device management solution? Perhaps using something like Jamf Self Service? Because it otherwise wouldn't be possible to deploy any changes without that kind of capability.

[1]: https://docs.docker.com/desktop/cert-revoke-solution/

Their proposed solution didn't work on many of my fleet's devices. I had to go deep into a reddit thread + stackoverflow comment to figure out a proper mitigation for several devices, and it involved extremely invasive/terrible things to remediate. Their fix doesn't even work for everyone, and they're clearly prioritizing paying customers for that kind of support. Like, fine, but don't publish a status that says "operational" when your "fix" is so slipshod as this. Just absolutely embarrassing, in my candid opinion. I know enough to fix this on any device but it isnt just a singular problem local to myself, and they haven't exactly been communicative about the root cause or very helpful to anyone that isn't paying them a lot of money. Like, I understand its "open" source "free"ware and YMMV there, but this kind of thing is entirely preventable most likely. There are so many viable alternatives to docker desktop that this has finally made me jump ship. There's no possible way this isn't a sinking ship, in my opinion. The writing has been on the wall for a while.

Company zero trust policies can make that impossible. In such cases you can reinstall official approved releases but not manually sudo random things

I'm aware that some companies implement that kind of security policy. But that makes it the company's job to make sure fixes for software critical to development are rolled out in a timely fashion. The way the OP is raging at everybody except their own company is totally unreasonable.

Also, disallowing root is developer hostile security theater IMO. It prevents a wide range of debugging techniques while being limited in the kind of harm it prevents.

Obligatory XKCD: https://xkcd.com/1200/

It’s theater but any heavily audited environment will have compliance requirements like this. It is unreasonable to blame the person implementing the requirements as if it’s their fault the compliance stipulations exist.

and even then it’s completely reasonable to disallow unlimited privileges on a managed workstation. the first time you deal with a breach because some idiot developer decided to install malware masquerading as a chatgpt wrapper will teach you this lesson.

used to have a gig where local dev worked on all machines except mine lol if it is only broken when Docker messes up horrible but not the worst it could be if perspective helps in this situation :)

Not having sudo on a work managed mac is so completely common that I struggle to take anyone seriously that argues otherwise and I'll just leave that comment here

My sample of 3 shows the Macs in big corp has some way to temporary get sudo. Some sort of proprietary app you run that grants you permissions for a period of time. Sometimes there is a text field for "reason", but no one ever reads it.

A work managed mac without a sort of helpdesk that can help you fix an update issue of approved software is equally unserious.

I have never had sudo on a work managed device, but always had a phone number to call for exactly these types of issues. Explain the problem, point them to the fix. They look it up internally, call me back half an hour later, take over the machine and perform the procedure.

Is it frictionless? No. Is it impossible? No, just part of dealing with corporate IT.

Imagine I am the corporate IT here. I don’t even need docker desktop! I’m switching to something else. This is now a huge unnecessary pain that still has yet to be explained.

Turn off updates, test things first

Docker has jumped the shark as a company, good thing the alternatives are maturing quickly. We've had pretty good (not perfect) success with the Colima project.

What did I miss?

They decided they want to actually get paid for their work and changed their pricing structure so large companies need to pay for their software and apparently that's terrible and everybody started being very vocal about alternatives despite still being able to use it for free.

sounds reasonable

Heads up to anyone at a company using Docker Desktop:

If your company has over 250 employees or more than $10 million in annual revenue, it's against their license terms for anyone at the company to use Docker Desktop without paying for a subscription [0]. Docker can and will go after companies who are out of compliance with this (ask me how I know).

If paying a subscription fee for a sub-par product doesn't sound great to you, I encourage you to check out one of the many alternatives available. I've used Colima (MIT licensed) [1] for a few years now and it works great.

[0] https://www.docker.com/pricing/faq/, see the section "Who’s required to pay for Docker subscription plans?"

[1] https://github.com/abiosoft/colima

I'm hearing about lots of alternatives lately: Comila, Rancher Desktop, Podman Desktop, OrbStack, etc.

Which one do folks like best?

I'm partial to Rancher Desktop, in my experience it's the most drop in replacement.

If you find yourself bottlenecked on IO performance, OrbStack is the best. Also a fan of the fact that it has a very similar VM function to WSL2. Also supports using either Rosetta or Qemu for emulation of Intel architecture, and has some kind of Kubernetes integration that I didn't try.

Last I tried Podman Desktop, it was OK, but had worse performance than Docker and was a little rougher around the edges. Still, not bad.

Colima is probably fine, but I had issues that were showstoppers for me. Trying to get it to consistently start up correctly in a CI environment seemed oddly challenging, and it doesn't seem to support IPv6 at all. YMMV.

Have not tried Rancher Desktop.

I did evals of most of the popular ones for my company and my opinion is OrbStack is best performance and most reliable. The Kubernetes integration is also very nice and resetting it takes about ~10s in case you get in a pickle.

OrbStack has been great. I'm not affiliated with them but a happy customer. They've had issues where an update broke them in very _weird_ ways, but I've been very impressed with their support engineers on this.

I've even had a situation where the support engineer jumped on a call with me to see the repro as it was a very specific/hard to share a simple repro situation.

Big fan of colima. It's pretty fast (in some cases not on parity with Orbstack), but it's free and doesn't have annoying Electron GUI or any crap like that.

Orbstack would be my next choice, but costs money for comercial user and my employer is a tight arse.

I experienced this earlier in the week. Didn’t realize it was an upstream issue. I ended up re-installing docker which somehow fixed the issue.

Work machine is managed by orgs fleet management software (jamf?) and thought it was an issue on their end (bad conf push).

We're relying on buildx bake. And few days ago Docker just stopped working. I tried the steps that they outlined with no success.

I know I can still run it from CD/CI, but I don't want to jump around like a crazy monkey trying to debug my builds locally. I had issues with docker desktop just hanging on my mac.

I've been using podman for over a year with docker context pointing to podman so I can still use bake.

I've moved some of our workloads to s2i and bash scripts and could not be happier. Old good sysadminy stuff just works.

I had a bizarre issue with Docker Desktop on Windows this week where it had somehow been uninstalled. I only occasionally use Docker, probably used it about a month ago (with summer Christmas break here), but when I tried on Monday it just wasn’t on my computer any more.

My PC isn’t managed at all by any other system or person.

These are usually caused by failed autoupdates.

It uninstalls the old one, fails at installing new version -> whoops, gone.

We ran into this, informed our users to reinstall Docker (we use Homebrew), and seemingly have had no further issues. I guess we're the minority?

I would be more inclined to blame this on Apple than Docker, but I haven't looked too deeply since things are working for us. Curious about you folks' opinions.

Colima on MacOS is so much easier and better than Docker Desktoo

Been using and happy with Podman Desktop for a while now on Mac

Podman Desktop on Mac uses weird user-pace IP stack which does direct socket allocations on the MAcOS host itself.

Additionally, Podman Desktop on Mac always return on ICMP echo, try running `ping 5.5.5.5` or any other non-ping-able IP and see it yourself.

Can you elaborate on the direct socket allocations?

You can see `tcp` sessions of the container using `netstat` on the host MacOS if you use Podman Desktop.

That's not the case if podman runs on a Linux VM on UTM for example. Additionally, ICMP behaves correctly in the later case.

https://orbstack.dev/

How can a broken update like this even happen? Surely Docker goes under rigorous testing before actions like this? I'm still waiting for a detailed post mortem, [0] but quite frankly this is a very basic intern mistake, but done by "senior" engineers.

In the mean time, I'm using Orbstack. [1] Much faster, lightweight and a native alternative.

[0] https://news.ycombinator.com/item?id=42695066

[1] https://orbstack.dev

I really like Orbstack, switched to it and never looked back.

+1 OrbStack is incredible

Also odd that it's taking so long to patch. A week is a long time to be without a key part of developer tooling.

Their signing key was revoked by apple. Unclear if this is by accident or in response for something. But the downstream effect is that it breaks updates.

is there anything like firecracker that works on macOS/Linux/Windows? or does anyone know if someone is working on one

OrbStack

is the best alternative to Docker Desktop on Mac.

When Enterprise pricing is "contact sales", I usually just close the tab.

I never needed Enterprise Docker Desktop.

I hardly even need Docker Desktop.

It just proves useful to start and stop Docker this way, since it uses fewer resources.

Thanks for posting this!

Yeah I was having some trouble with it a little less than a week ago.

I downloaded the new update and then my app was corrupt and couldn't open so I just re-downloaded and installed and it was fine.

Orbstack

Reply on news.ycombinator.com