Hacker News Reader

Comment by bjt | Hacker News Reader

I remember at the time there was a "Silicon Valley rides to the rescue" subtext. Though a Silicon Valley person myself, that rubs me the wrong way.

I appreciate the author resisting that framing, and admitting that on that first day, the contribution of the new team was basically to ask "Can we install New Relic on the servers?" The response to that question really illustrates the root of the problem, in my opinion. It's not that the people who built the original healthcare.gov were stupid people. It's that they were operating under constraints and incentives that emphasize rules and compliance over making the whole system work well.

View on HN

Bit of both, really. We got https://en.wikipedia.org/wiki/United_States_Digital_Service out of the original Healthcare.gov disaster; if you've used login.gov I suspect you've found it one of the nicer government websites out there (for example: it's one of very few to explicitly advise against SMS 2FA, and to offer U2F support), and it owes its existence to the USDS.

Not just USDS, also 18f -- https://www.usds.gov/report-to-congress/2017/fall/login-dot-...

I was so pleased when I learned that login.gov implemented U2F and advised against SMS.

They also make websites which have excellent color contrast and intuitive design, and you can tell that a lot of thought went into each one.

The design system is really well documented: https://designsystem.digital.gov/

I made a template repo for getting started with next.js and the USDS when I worked on Paul's team: https://github.com/adhocteam/uswds_nextjs_starter

> It's not that the people who built the original healthcare.gov were stupid people. It's that they were operating under constraints and incentives that emphasize rules and compliance over making the whole system work well.

Great comment. I spent the first half of my career in classical "silicon valley", direct-to-consumer businesses. Lately, I've been working in software in highly regulated industries. The differences between the two cannot be overstated. Even if you look past tiny (/s) issues like "you can't lose people's money, ever", you've got reams of often-conflicting regulations at the local, state and federal levels. Making a small text change to a webpage can require multiple lawyers.

This kind of well-intentioned stuff sometimes makes it impossible for the greatest engineers to do even simple things quickly. Work at the government level, and it gets worse, because now you have politicians reacting to your every change.

It's not that the people who built the original healthcare.gov were stupid people

Thats very kind of you but they probably were, if not stupid, then just large companies who routinely do these big government websites and mess them up and only just get them done years late and way over budget and don't really give a shit or face any consequences.

This NPR article says the original site was built by a bunch of different companies, most of them subcontracting out to other companies, and no-one really in charge of the overall architecture apart from some little subsection of Centers for Medicare and Medicaid Services who probably weren't ready for the cat herding required.

https://www.npr.org/sections/alltechconsidered/2013/10/25/24...

I can tell you that the difference between a site made the 'Silicon Valley' way and an 'enterprisey' site made by EDF or Acccenture or Fujitsu or one of those other big companies is a massive difference. Its a completely different world.

USDS, 18F, and the Digital Services Coalition have made remarkable improvements to how digital services are delivered for government.

It's quite improved, IMHO, especially from where things were in 2013.

There are still some shitty things out there, like the National Parks lottery, but things are improving.

USDS, 18F, and the Digital Services Coalition have made remarkable improvements to how digital services are delivered for government.

Right, and thats because USDS and 18F kick out all the big old 'enterprisey' consultancy companies and bring it more in-house. And presumably the smarter version of in-house where you hire people who can actually do stuff. Digital Services Coalition sounds like a sortof in-between where companies can apply to join but presumably they are finding ways to avoid the crap old enterprisey ones.

Indeed.

Silicon valley did ride to the rescue.

Obama brought in Mikey Dickerson, head of google SRE, as well as several top people in Facebook (at least those were the people I had directly heard about) and humbly said "we failed, we need your help." From what I remember hearing he literally brought top valley people into the oval office and gave a "your country needs you" speech.

"Install new relic" really minimizes what was actually happening. "Can we install this tool that helps us understand how the system works" sounds a lot more enlightened doesn't it? Understand what's broken is the first step to fixing something and that's exactly what "installing new relic" does. If you don't have a monitoring system, that shows an extremely critical failure of technical leadership. It means no one knows what's happening. No one is measuring how things work. The google SRE book devotes chapters to monitoring, probably 25%+ of the book is explicitly about monitoring.

If you listen to Mikey present his work on healtchare.gov, he does not emphasize the solution, he emphasizes the problem: No one knew how things worked, no one knew how the pieces fit together, no one could measure why the site was performing so poorly.

The step after "installing new relic" was getting 50+ contractors in the same room and making them justify themselves. The contractors sold themselves to the government, but there wasn't a person on the government side with enough understanding to know you don't need 3 different load balancers at the same level for the same traffic and every system does need monitoring. How many different data stores do you really need?

If you listen to Mikey Dickerson give his presentation live, to the people he hopes to recruit, then it was very poorly managed at the technical level. Were those people stupid? probably not, but they were giving out lucrative contracts to any company that said they could help. So the makers of the original healtchare.gov (and by proxy the American public and Obama administration) were being taken advantage of for their ignorance.

> It's that they were operating under constraints and incentives that emphasize rules and compliance over making the whole system work well.

It's that trust was given to American companies who directly benefited from taking advantage of the government, and not to individuals who work on behalf of the public with specific expertise.

Mikey ends his presentation with a powerful call to action. If you aren't going to help the government with your own expertise, and make that sacrifice on behalf of the public, then some contractor will happily fill that role.

> Were those people stupid? probably not, but they were giving out lucrative contracts to any company that said they could help. So the makers of the original healtchare.gov (and by proxy the American public and Obama administration) were being taken advantage of for their ignorance.

Good points but that sounds to me like too simple a story. The people managing healthcare.gov weren't working their first day or managing their first project; you don't get that job without some experience. And if you have some experience, you know what you don't know, you know how to deal with those situations.

(I'm not implying some other specific story - I don't know what happened. I'm just wary of such simple answers.)

I saw Mikey Dickerson talk about this at the time, and he was very explicit that the engineers on the project were all perfectly capable, but they were hobbled by an IT services culture that made them all terrified to be the person blamed for any mistake.

He also said there was no culture of real-time monitoring. They literally had a TV running CNN, and whenever the chyron said "healthcare.gov is down again" they would spring into action. Amazing.

I can't really phantom that. When I was six grade or something, I had old PC with Debian in my room as "server" to play around and serve some random stuff from. And I had real time monitoring on that.

The cultural explanation makes more intuitive sense to me. And the lack of monitoring follows from the culture - they don't want to know.

Exactly this, it can be so frustrating for teams, when they've been repeatedly trying to tell people what the problem is, to have some outside expert come in, point out the same things and get them green lighted.

Reply on news.ycombinator.com