Alaska government spent $28M on database project before it was scrapped
adn.com
adn.com
I always wondered why they take on such small projects. Northrop just botched a system in Tennessee[0].
http://www.tennessean.com/story/news/2015/01/12/tenncare-fir...
I've been programming recreationally since I was a kid in the '70s, and I've started to do some professional work in the last few years. I started out in BASIC, and then became familiar with C, Fortran, and Pascal in college. I've since learned Java, and now my language of choice is Python. I have a basic competence in SQL, git, and any number of other tools we use in modern development approaches.
I know work like this isn't easy. I know the data is stored in esoteric structures on long-outdated hardware, running on OSes I've never touched. I know there are issues with security clearances and background checks. But I have to think - give me $28m, let me work on this full time for a couple years, and I'll have some meaningful work done and some usable end products. I'd give myself a modest salary, and hire people with way more experience than myself in the areas needed. I can even see a path into this line of work - complete a series of successively more complex and sensitive data-migration projects. I like to think that doing that work wouldn't turn me into someone running a firm like the one hired in this situation.
For people who've worked on projects like this that turned out well, is this something any of us with reasonable competence could get into, if we build an appropriate portfolio?
Also, I live in Alaska so I find this story particularly frustrating. Not just because of the wasted money, but also because I want our law enforcement personnel to have a well-functioning system at their disposal.
It's not even the 80/20 rule. It's more like 99/1 rule. As in, I see this requirements list, and I have two options for you. A) I can deliver 99% of what you want for 1% of your stated budget, or B) I can take 1000% of your budget and still fail to deliver 100%.
High quality code = rapid development.
There is little technical challenge in a product like this. Building crud apps that scale to low tens of thousands of users is not that challenging. Since (I believe) we all share that belief, then why do projects like this keep failing?
I've worked on large medical information systems. The problems those encounter are
(1) you essentially never get to bulk replace entire systems; you will have to support ongoing data ingestion and export to other running systems, often systems not designed to support this or with poor apis;
(1a) if / when you fuck up, critical processes are disrupted. You have to engineer your systems and create backup processes to make this very unlikely to happen. There is a case I can't really share too many details of where a database was accidentally dropped in a datacenter of a hospital chain you've heard of. Said hospital chain was unable to access medical records for 20+ hours; this was in the 90s running on giant hpux machines and it took a long time to reimport data off tape. My employer was on the hook for $30M iirc, and I wouldn't be surprised if this actually contributed to serious patient injuries or deaths. Now obviously law enforcement isn't quite as critical as medical records, but it's not far behind.
(2) over the life of such systems, lots of workflows / processes have developed around using them inside hospitals or large doctors offices. You have to discover these processes, mostly stored in the heads of line level workers and their immediate supervisors, and figure out how to enable them in your shiny new system. To be concrete: the medical record system didn't support some requirement. That didn't stop users from needing it. So employees or doctors decided to use a field in a certain way, and they know how to interpret the presence and/or data in that field. You don't even know a field is being repurposed, let alone how they actually use it. This will be a large problem importing all the old data and in recreating the necessary functionality.
(3) no one person, and probably no small set of people, know or understand all the ways the system is used. Somehow, you have to capture that knowledge and use it to create a functioning new system. At least the medical record company I worked for hired lots of doctors and nurses out of hospitals to help them understand how users used the system in practice. You essentially don't understand at all how prosecutors, police, secretaries, and all the other people involved in law enforcement use this system.
There are clear parallels to replacing systems like this. Particularly when you have to do a live transition.
I'm not saying I could do a perfect job - just that I imagine I'd have something to show for working on this problem full time for 10 years, with the stated level of funding.
But the challenge in these projects is generally not technical, which your post focused on. It's all management: management of scope, management of requirements gathering, management of expectations, management of deliverables. Management of having to work with people who, frankly, are dumb, and for whatever reason resent you, but they use the system and you have to figure out what they need anyway.
Oh, and the better question in all of this is why is Alaska building such a system? Are they such special snowflakes that not a single other state or country in the world has a similar system that they could buy and minimally to moderately adapt for local needs? Forty-nine other states, plus DC, plus puerto rico, plus the fbi's system, not to mention the provinces in Canada or states in Mexico. Surely someone has a system that could serve as a 90% starting point.
Relatedly, why aren't these systems mandated to be open source? The public ought be able to inspect the algorithms that regulate them, and various parts of government ought be able to collaborate freely.
Even with the hand-holding and guidance offered by Django and Rails, you still get low-complexity CRUD business apps that are drowning in bugs.
As far as the people who are actually in charge are concerned, the important outcome is to spend (thus justify) their budget, and to be seen to take action.
As for selection, it's about shmoozing the right people, who you know, and navigating labyrinthine government conditions that have more to do with bureaucratic ass-covering and factory production schedules than software.
One clue to why if failed describes there being a tension between departments and a new governance board. Providing a system like this based only on best practices should be pretty feasible if you throw out all the petty politics and "perceived but useless" pet features.
I'd imagine the current system (as it's mainframe-based) pretty much follows the typical forms entry (perhaps they're still using green-screens?). Moving the existing system as-is to a modern version of an RDBMS with an equivalent web-based front-end should be the first step. Once you're current, you can add features as you go.
Money is always subject to whomever got elected last November and a great many people resent you. They're 57, been working with the previous system since the mid 80s and they like it dammit. They see the writing on the wall and the mainframe operator or data entry specialist job they have is a goner. Shy of a full pension too.
Or the prima Donna department that always got their way is blue balling the analysts the whole way through because they don't own this anymore.
Problems in this type of projects are all people and processes. I also work in Healthcare IT on the vendor side and I see projects that get shit done and project that die. It's about management.
My strategy is to tell people "I am going to make it work and you are going to either help me, or watch helplessly as I make it work without you and make you look like an idiot".
It helps to take assertions like "you cannot do that" literally, even though they tend to be said to mean "you may not do that" or "you should not do that", and prove that you can in fact, do that all you like because there are no laws of physics preventing it.
It has caused a few surreal meetings, like the time when a guy did a 40 minute long presentation on "I need more people and more money in my workgroup to make this work" while the first batch of completed, working, tested production units (built by me and one other guy after we got sick of the BS, I even paid for the manufacturing out of pocket) was in the middle of the conference table.
Imagine a half dozen people trying to ignore what was literally in front of their eyes.
Another time it also has caused me to escort a security guard out of the building because he was being disruptive. He had been told to escort me out of the building, grabbed me without asking my permission, and found himself in a headlock, then was informed that if he did want to fight me, he was welcome to. Then I got back inside and finished my business.
Sometimes you just have to say "full speed ahead and damn the torpedos", or nothing gets done.
(My father's rule of fights: Never hit first, always hit last)
Unless you and those hundred developers had previously formed a company that was building software for the government, you wouldn't even be considerable, as a matter of law.
The federal government has a list of regulations they're subject to, specifically documented in the "FAR" (Federal Acquisition Regulations). One of the first requirements is that they cannot source work from a vendor who has not done work for the government before, and that they must be able to be recommended by the agencies for whom the work was done.
In the industry, these are referred to as "quals" (short for qualifications), and it means, specifically, that you have done this kind of work for a previous agency and done well enough that the receiving agency would recommend you.
The only way to "get" quals if you don't already have them, is to sub-contract with a qualified agency, like SAIC, CACI, Boeing, etc, and get similar work so that you can gain enough qual to do the work yourself as a prime contractor.
Generally, by the time you've done this, you'll realize that no, you and a hundred good developers are not enough to deliver a project of any size to a reasonably sized government agency, at least, not by yourself.
For a project that would require 10-100 developers, you'll need at least two architects, a contracts attorney, a strong project management team, and a team of experienced writers in order to even put together the proposal response enough that would make it winnable.
Sunlight Foundation tried to do this same thing, and ended up with a contract that wasn't even considerable by the agency. I cannot stress enough how difficult a contract proposal is to write, and how much work and effort is involved, and without a really good proposal, you literally could not even give your work away for free, even if you ignore that 90+ percent of government contracts are indeed "wired" for an existing vendor to win them. To even write a decent contract, you'll have to have a really intimate knowledge of how that agency is already doing things, which means hiring some people from within the agency for consulting before you can even begin to write a proposal.
On top of that, your architects will spend all day in meetings trying to explain away stupid things to competitors like how creating a new table in an existing database will not subject the agency to more risk of hacking, how switching from SOAP to JSON will not make systems more vulnerable, etc., etc., to an audience consisting of competitors who want you to fail, managers who want you to fail, end users who want you to fail, and other folks who are paid to disbelieve you.
Moreover, you'll be limited to doing all of your development on Sun Solaris (or AIX, or whatever) using Oracle Database, communicating messages via SOAP, etc., etc., and you won't be allowed to introduce Ruby, or Scala, or ZeroMQ, or whatever new-fangled tools you are used to using except those that already exist within the environment.
Reading this back to myself, it seems awfully pessimistic. It probably is, but that wasn't my intent. But, there are reasons that government projects cost what they cost, and while definitely plenty of it is wasteful, that waste isn't in there by design, but is in fact a product of a system designed to not generate the very waste it creates.
Your more plausible statement is that my architects would spend all day in meetings. That's why I proposed building a direct replacement (which can be specified from the current operating system as a first step). I realize you need to bypass that kind of bureaucracy.
EDIT: I think it's highly unlikely any government agency would take take an offer like this ... the risk for their career as bureaucrats would be too great. Unfortunately that means we'll never find out what a good team could do.
That said, my response was tailored towards federal government, as I'd completely lost the plot in which Alaska was the subject, so it's probably worth throwing a couple of caveats into my response for that reason alone.
big ticket projects for the FBI? Like VCF?
http://www.washingtonpost.com/wp-dyn/content/article/2006/08...
> the system delivered by SAIC was so incomplete and unusable that it left the FBI with little choice but to scuttle the effort altogether.
Look, I am okay with the contractor subcontracts to 34 other contractors. I will take that as hiring consultants from 34 other consulting agencies. But I can bet with my life they were all working in vacuum and in a full water fall model. No doubt we need to start with analysis, and propose a possible solution like database schema and migration, but one can still work with other parts of the project iteratively, e.g. front-end and some other micro-services. I also suspect contractors hire offshore which can further complicate the development process as one has to deal with language barrier (kind of a minor issue) and timezone difference. FYI I do work with offshore day-to-day and the one I work with are excellent.
Like healtcare.gov, and the fictional (but based on real life stories) Phoenix Project, there was little to no human resource and IT capacity planning, and management oversight is completely broken with 35 contractors involved.
I cannot underestimate the complexity involved in migrating old data, but I bet if we gather a bunch of database professors they can probably come up with a pretty good database migration plan and a new database schema design.
More than once I've seen stealth mode side-projects run on a single engineer's own time out-deliver and out-perform the fully funded project because the standard approach is to take absolutely trivial problems and heap on absurd layers of complexity which of course are full of bugs and utterly fail in the field.
I wonder if taxpayers have ever tried to sue for damages, but I'm sure it's impossible.
eg oregon is suing oracle over their failed implementation of the state obamacare site
http://www.reuters.com/article/2014/08/22/us-usa-oregon-orac...
I wouldn't call hc.gov "trivial" -- it started with all the fun of a massive-scale website, added X12 EDI workflow, then finished up with real-time communications with high-security government systems never meant to be interoperated with and at least a few of which probably can't communicate over red-black networks. Then they started adding complexity to it.
Admittedly, some of those added complexities caused major headache, such as the requirement to use ORCL IDM (which, if the hotwashes are correct, proved a constant Achilles' heel to the ops side -- who the hell put that requirement in the spec?), but the project started out high-risk.
The problem with government projects by the way, isn't with "project managers and engineers;" it's with the dozens of agency stakeholders, half of whom are political appointees and the other half of whom are waging bureaucratic turf wars against everyone else. With high-profile projects like hc.gov, you also get politicians involved, so everyone starts freaking out about things like the fear that some illegal immigrant working in a Kansas poultry slaughterhouse could accidentally qualify for health insurance and end up as the lead story on FOX News. And, of course, even though the seventh floor (and, in this case, the Oval Office!) gives lip service about how critical it is that the tech works perfectly, you'll never see an appointee or an SES employee actually try to keep the ship upright.
After that, add in large-scale government contractors that operate on a high-risk version of waterfall that requires every single piece to come together perfectly, on time, without errors, and for the internal specs and contracts to not shift at all -- well, it's a recipe for a massive clusterfuck, every single time. There's a reason Medicare is still running on mainframes and FERS has never gotten off paper -- the risks of failure are just overwhelming.
http://www.nytimes.com/2012/12/09/technology/air-force-stumb...
http://www.theguardian.com/society/2013/sep/18/nhs-records-s...
http://cdn.itconversations.com/ITC.IEEE-FBISoftware-2006.06....
TL;DL No one actually sat down and specifically told the contractors what was needed because they didn't bother to even work it out for themselves.
So the contractors could get away with what they wanted.
http://www.nytimes.com/2013/11/23/nyregion/3-found-guilty-in...
Multiple large organizations on a large scale consulting engagement seems to lower the chances of success.
"Charles Collins Jr., the lead author of the MTG report, said in an interview that the consulting group had encountered SAIC in the past. He said he believed this to be MTG’s first critical report on the contractor."
Their review of another department (Division of Statewide Services) was particularly damning though:
"In the December 2013 “organizational review” of the Division of Statewide Services, also conducted by MTG Consulting Group, auditors found evidence of what they called “a seriously dysfunctional IT support organization.” The review found that the division lacked governance and communication, its benefits to other divisions and departments were unclear, and there was a serious aversion to risk or change."
Sounds like the place was being run by Mordac: http://goo.gl/o7JRvr
The downfall of many a project. Most just don't accumulate $28m in costs before they fold.