Any service that has the potential to attract state actors, or attackers with access to frontier AI, MUST have a threat model that trusts no single employee or workstation.
From there the spec I described writes itself. No one can ever have the ability to mutate a system once launched.
A new server spec should be defined in a signed change, it is approved with a signed review, it is compiled deterministically from source and signed by multiple independently controlled systems which both must match, and then given multiple signatures. Then any engineer can deploy those immutable images, other engineers can remotely attest the expected PCRs from the independent signed builds. Finally secrets can be released to the machine by an HSM that validates m-of-n approval signatures.
This is table stakes zero trust deployment and again, the bare minimum.
Humans manually sshing to production machines and running commands is a 90s way of doing sysadmin that is exactly why Binance was just compromised and a thousand orgs before it. Survivors bias is not a reasonable security posture.
Also with even nested virtualization on modern hardware the perf hit is minimal, though distinct servers for each workload is fine too. At -least- sandbox each workload with something like gvisor at a minimum. Linux kernel namespaces are not enough as evidenced by endless exploits.
I practice what I preach here too. I spent the last decade building this way for high risk orgs responsible for hundreds of billions of dollars in value. I also founded https://stagex.tools and co-founded https://distrust.co and https://caution.co to make it easier for others to follow these basic tactics. Much more to come soon on this. All FOSS of course.
We need adoption on these tactics to be as wide spread as TLS if the internet is going to survive an army of AI armed skiddies.