They are one of the last companies we should be trusting with hosting infrastructure.
Add to that aging and wildly insecure ssh-based centralized ruby based infra management solution and I can only conclude everyone invovled has been asleep at the wheel on industry best practices for a over a decade.
Production servers should be immutable once deployed, remotely attestable at all times, with quorum signed secret management that trusts no single admin, with every workload in a dedicated vm with only a kernel, reaper, and target daemon, under an inaccessible offline base OS with TPM gated full disk encryption that fails closed if the system is ever modified after first boot.
And all of that is the bare -minimum-. Unfortunately negligence is industry standard since security engineers that actually know how to do system hardening are crazy rare in the bay.
The most grating thing is requiring every workload to be in a dedicated VM. You are throwing so much performance away. What kind of threat model would justify this? Usually there’s no threat model analysis at all.
Banning opensshd is again mostly a fool’s errand. It’s much more productive to design a system that writes the authorized_keys file read by opensshd. For example you can design a system that requires peer approval before your own key gets written to authorized_keys.
From there the spec I described writes itself. No one can ever have the ability to mutate a system once launched.
A new server spec should be defined in a signed change, it is approved with a signed review, it is compiled deterministically from source and signed by multiple independently controlled systems which both must match, and then given multiple signatures. Then any engineer can deploy those immutable images, other engineers can remotely attest the expected PCRs from the independent signed builds. Finally secrets can be released to the machine by an HSM that validates m-of-n approval signatures.
This is table stakes zero trust deployment and again, the bare minimum.
Humans manually sshing to production machines and running commands is a 90s way of doing sysadmin that is exactly why Binance was just compromised and a thousand orgs before it. Survivors bias is not a reasonable security posture.
Also with even nested virtualization on modern hardware the perf hit is minimal, though distinct servers for each workload is fine too. At -least- sandbox each workload with something like gvisor at a minimum. Linux kernel namespaces are not enough as evidenced by endless exploits.
I practice what I preach here too. I spent the last decade building this way for high risk orgs responsible for hundreds of billions of dollars in value. I also founded https://stagex.tools and co-founded https://distrust.co and https://caution.co to make it easier for others to follow these basic tactics. Much more to come soon on this. All FOSS of course.
We need adoption on these tactics to be as wide spread as TLS if the internet is going to survive an army of AI armed skiddies.
All sorts of intel you can get about all sorts of companies that way.
That is part of the entire problem in the industry. No standards and no accountability. Those with the purse strings only invest in security that is marketable or protects IP. Anyone that invested in proper remotely attestable provably quorum managed systems is going to be bragging about it and happy to show off the proof to earn trust from their customers. I do.
When no one is bragging about doing these things, it is because they are not doing them, and they just have a bunch of humans running around with god powers to execute any code they want on production systems or in binaries that will end up on your laptop.
To get a clue about their internal practices, lets see what they do when everyone is watching in public.
https://github.com/apple/foundationdb/commits/main/
No signed commits, no signed reviews. Merges done via browser. One Github API token away from being supply chain attacked. Also put one of those recent usernames into "forgot my password" and you can verify they do not mandate hardware 2FA. One compromised email account and you get to make a PR as a rando, then merge it with an Apple account and huzzah, you now have code in a critical repo used internally at Apple. Also releases are not signed or reproducible and are just cut by github actions. Another API key you could use to be even sneakier.
Almost any community maintained Linux distro does way better than this. Inexcusable for an org of their resources.
If this is how poorly Apple handles supply chain integrity in public, why would anyone believe they do any of this internally. I have also talked with multiple current and former Apple employees close to such things off the record but obviously will not out which ones.
What better evidence could I possibly provide?
Also see the other commenter in this thread who identified themselves as a former Apple contractor admitting Apple uses Jenkins. -Jenkins-.
Scroll up, I asked for evidence of this specifically. This is a pretty damning allegation and worthy of further examination given the security implications for many, many users. Frankly, whatever they're doing in unrelated public repositories is concerning in a different context but otherwise thoroughly irrelevant to this claim above
https://jobs.apple.com/en-us/details/200440680-3337/senior-l...
Puppet does not have quorum signing or approval support. By -design- it means one god server that ultimately trusts every ssh key of every admin to make changes to a fleet of -malleable- RedHat servers via ssh like it is 2005.
But for just one attack chain to humor you:
1. Malicious party gets access to one Apple engineer ssh key (Or literally just their email account since they sign nothing and the key could be swapped out temporarily)
2. They make a PR injecting a subtle malicious dependency to any of the core Apple component repos almost certainly used on prod servers on the Apple github.
3. They use their stolen key to merge the PR.
4. Code lands in next release and eventually lands in Apple production servers because nothing is signed so how would anyone know?
We learned nothing from xz.
Think of it as a predecessor to Facebook, plus whatever training platform you like. One of these guys wrote a complete software simulator for the iPod Touch-based device they were using in all the stores to handle point-of-sale transactions, so that he could include that in the training.
Content was created on macOS. Content was consumed on iPhones and iPads. The backend servers ran on Linux. It was my job to get all that compiling on the Jenkins server sitting under my desk, instead of on their laptops. I had to make minor changes to each of those code bases to get that to work.
Their build processes were quite advanced for the times. Thousands and thousands of regression tests. Thousands of UI conformance tests.
Then I had to get that Jenkins server up and running on the in-house VMware based cloud they had, where they sucked up their entire last two runs of rackmount xserve hardware and turned it around for their own internal use.
I was given a VM and remote access, and then that was that. My job was done. My contract was over, and I moved back to Austin. But that was a few years ago.
Thank you for having the balls to admit this in public.
Hopefully others can stop putting Apple on a pedestal as some gold standard for high accountability and rigorous security.
And for that matter, if you’re running your own CI/CD solution, Jenkins isn’t that bad, either.
Also mind that this work was done about a decade ago, and Puppet and Jenkins were newer and shinier tools back then.
Now, if you can operate in the cloud, then I can point you at solutions that I think are better — depending one which cloud you’re in, and what tools are natively available there. But on-premise hasn’t changed all that much in a decade.