Comment by _bnmd | Hacker News Reader

Comment by _bnmd | Hacker News Reader

I feel Nic's pain. Here is the original article about the talk he gave before leaving: https://www.airforcemag.com/air-force-leadership-chief-softw...

> One of Chaillan’s main concerns is incorporating security into software development, a practice known among IT professionals as DevSecOps. With a lack of basic IT infrastructure, implementing DevSecOps has proven difficult, he said. What’s more, there has been some resistance among those used to the more traditional approach of considering security after development and operations.

We're standing up basically everything ourselves from scratch. The mandate was basically "we have a critical need for a new capability. Here is an AWS account and five developers, so make it happen." That's it. So everything from standing up CI/CD pipelines, to building out a cluster, to configuring storage and networking, to writing and testing the application code, to maintaining environments and deployments, is falling on us, with no support.

I'm not going to say what the product is for reasons of OPSEC, but it is inherently a product that has extremely high security needs. Yet in the rush to be able to tell some high-ranking people we have put an "MVP" in production, we've skimped in every which way it is possible to skimp. I am aware of so many holes in the system, but Air Force pen testers didn't find them, so our product manager is being pushed to go forward and we'll worry about security later.

To my mind, this is absolutely unacceptable for a critical defense system, but nobody is asking my opinion. Supposedly, we keep being told we'll lose funding and get the plug pulled if we don't hit some important milestone at some exact date. By being "agile," we can deliver a broken, insecure "MVP" and "iterate" on it until we have a real product that actually meets its requirements.

You can't do this crap with defense systems. This isn't Etsy. Deploying broken shit has far different implications than when all the exemplars from the DevOps Handbook do it in order to find all their bugs in prod and turn their users into beta testers.

I can echo these concerns having been a contractor working on applications in more than one of the DoD cloud platforms, including CloudOne (a subset of PlatformOne, for readers not familiar with the flurry of DoD cloud offerings that have sprung up over the last few years). I recently changed jobs in no small part because of the massive incompetence on the USAF side. It’s really quite stunning. My entire schedule was eaten up by unnecessary meetings where wholly unqualified USAF officers (current and retired) in PM or similar roles would pontificate endlessly about just absolute nonsense concerns. Like hours of arguing about how to label a button on a form. Constant bike shedding. No users involved meaningfully in the feedback cycle. And I swear all these old USAF guys just straight up hate their users. They will suggest the most user-abusing possible design because they think their users are stupid and need 10000 confirmation dialogs to avoid making mistakes.

And on the legacy non-cloud side of things… it’s a horror show. No CI/CD. No testing (a lot of my job was bolting awkward test harnesses on to existing legacy software to compensate). Inconsistent and ever changing project management systems (they switched from TFS to Jama to Azure DevOps to Jama again and then when I left were talking about moving to JIRA. Our cocontractors were insanely unqualified. They were really proud of how cutting edge they were for adopting git for VCS. In 2019. It’s crazy how bad all of this software is, but at least it wasn’t on some internet connected server before.

I would wager that Etsy (and most big cloud-native unicorns) probably has far, far, far superior infra, SW & ops in place than just about any gov agency... and the ones that don't (Zoom) get called out and are forced to fix it.

I don't think you'd find many people who'd argue with that. But Etsy's requirements are far different from a lot of DoD systems.

The requirements may be different, but the end result of the DoD project may well be worse (and not meet all the requirements, but somehow still tick all the boxes).

I'm not at all trying to crap on Etsy. I'm just saying these stories about industry leaders that are deploying to prod 100,000 times a day are able to do this because they own applications that can break and it's not really that big a deal. HBO Max, Prime Video, and Netflix pretty consistently just crash every now and again and reboot. Whatever. It comes back up and you can watch your show. Bizarre bugs are at worst minor annoyances. You can't do the same thing with a weapons platform control module. You need to test the hell out of it and know for certain it works, in every possible edge case. Releases to production are heavily gated for a very good reason.

You’re just banally pointing out that different applications have different correctness requirements.

DoD applications break all the time though!

Yes, even weapons consoles! I served on a submarine whose fire control system would literally crash and reboot if we encountered a situation that was especially frequent for the class of boat I was on. We worked around it, like the users always do, but please do not confuse 'DoD unique requirements' with 'Etsy can't do this'.

> You can't do the same thing with a weapons platform control module. You need to test the hell out of it and know for certain it works, in every possible edge case. Releases to production are heavily gated for a very good reason.

Do you really think Etsy doesn't run tests?

Let me tell you as someone who has run reactor restart procedures a hundred times on a real live submarine, really under the water when the reactor was scrammed: the best way to get good at something is to do it a lot. Whether that's starting up a reactor, or deploying to production.

That sounds disturbing. However, that is how the military has done things in other domains for generations, and probably forever.

Remember that the term SNAFU came from the military; watch some WWII through Vietnam depictions of it: Before the modern era of its glorification, the US military was synonymous with absurd, screwed-up systems and policies that the soldiers overcame with chewing gum, duct tape, initiative and a sense of humor. (Some say the reputational change is due to the shift from the draft, which caused a wide segment of the population to be familiar with the military, to volunteer professional personnel, which results in most people having no clue about it.)

I'm not saying it's a good thing or that it shouldn't be improved, but the military (and every large institution) have always had a lot of that crap. I remember a Marine officer telling me that to never fly in one of their tilt-rotor aircraft unless I see a lot of hydraulic fluid on the ground - because if I don't, then it's out of hydraulic fluid. As they explained, they go to war with - their lives depend on - tools made by the lowest bidder.

> has extremely high security needs ... I am aware of so many holes in the system,

It sounds like you need to have a very serious talk with your management.

I really hope that leads to a resolution of the problem.

If not, lives and nations sometimes depend on these systems working, so you might be obligated to ensure that the right people understand the problem.

Security is clearly not incentivised in the organisation you are working for. But delivering on schedule and making your superiors look good is. So that’s the rules of the game you have to play with.

You should tell me what those issues are.

Honestly, best of luck, but it's mostly to do with PKI. The DoD PKI office won't issue code signing certificates for NPE users, so artifact provenance assurance either can't be automated, which means it can't be scaled, or we're forced to use self-signed certificates, which combined with having small teams where everyone is forced to have admin privileges because no one actually owns the infrastructure and provides timely support for change management requests, leaves way too many systems incredibly open to insider threat.

Unless my understanding is off, someone needs to build you an environment which satisfies these concerns relative to the different roles and boundaries. This is entirely possible.

Reply on news.ycombinator.com