From $63M to $600M? I would have thought that ppl may have noticed when the costs doubled or tripled the budget. Almost 10x the original budget? That type of oversight must go pretty high up.
From $63M to $600M? I would have thought that ppl may have noticed when the costs doubled or tripled the budget. Almost 10x the original budget? That type of oversight must go pretty high up.
Clearly fraud and kickbacks kept this stumbling along much longer than a normal failing IT project would. I'm not sure why or how an expense like that on an overage like that on a major payroll system would not be on the mayor's daily agenda. The right people must have been getting those kickbacks.
I've been dragged into 'feature acceptance meetings' that literally dragged out for weeks when testing of a new chat component devolved into a days-long discussion on whether the graphic in the header (provided by the customer) was appropriate. The unspoken issue proved to be that they didn't want us to use 'barren trees' in the picture of their building. We switched the photo used from one taken in winter to one taken in summer. This took something like 400 man-hours to resolve, after which we achieved signoff, but at the expense of having met budget.
Another instance on the same product was feature creep to the extreme. We had originally won a firm-fixed contract, meaning we would build products a, b and c for x dollars, and if it cost us more, it came out of our pocket. If it cost us less, we pocketed the profits. The customer decided, on day 1, that they actually wanted a lot more features than were described in the project scope defined, and that we won a legally binding contract to perform. Out of accomodation, we changed their product to a time-and-materials contract, meaning we had a fixed markup on labor and a minimal markup on hardware / tangibles. This then involved them dragging the project out for years longer than anticipated and signing invoices all the way, then griping at us on the budget near project completion and asking for a bunch of free labor (which is illegal under their new contract). The problem in that case was that the project management and senior execs changed repeatedly. We were at first working with their Office of Science and Technology, then with Information Services, then with Public Affairs. Each of these department changes involved swapping our equipment out so that we could have appropriately charged-back equipment to the appropriate department, which meant weeks of non-work. Each department had a different spin on what they wanted out of the project. We spent 8 months and thousands of man-hours implementing a very elegant services oriented architecture that could surface all of their legacy systems to a single web portal. Then we changed departments, and they literally told us to turn that feature off, and instead focus on a visual redesign. We implemented a single sign on service that we integrated with our product, but could be extended to work with literally every system in their enterprise. It was abandoned due to a department change.
During the first phase of the project, we'd worked with the customer to build out their data center. They wanted a new data center because the division that owned the project was going to be adding a bunch of other programs to integrate with what we were building. They spent over a year on construction, converting a service garage into a data center. They bought millions of dollars worth of EMC storage hardware and Cisco routers. Then they realized that they didn't have any budget planned for out-year maintenance, so they moved the venue. That was somewhere in the realm of $50 million dollars that was literally just thrown away, as the last I heard, they were re-converting the data center back into a garage.
In short, government contractors catch a lot of flack, because they control the visibility of these things. The contractors often can't speak to details of these projects due to security clearances or NDAs, and at the end of the day, the government is always happy to blame the vendor. It's well practiced, and designed with a variety of scapegoat clauses built in.
Replacing the logo itself took all of about 30 minutes (receive new picture, digitally retouch, overlay onto background image, resize, play around with transparency, save to png.)
Also, bear in mind, the 400 hours is 400 man-hours, which is a cumulative number based on the number of people involved. If you've never done business with government, let me assure you, it was a lot more people than necessary.
I recently did contract work for an agency and had instances where a simple spacing or image resizing took days of work because the client or the agency would understand and then forget that I must crop and/or resize images for them all to be of one size.
And then they changed it back to the original.
Your assessment of the software and services seems fairly accurate to me (there are a couple of exceptions: CAS is simple enough to be easily understandable and works ok, and Atlassian Crowd looks ok too).
... it's just the article notes a subcontractor has already been convicted of fraud and kickbacks, so this might not have been a project quite like the projects you describe (though if this is a Terry Childs-type case, I'd love to hear about it).
I meant to say those things, but got lost in typing so much other nonsense and trailed off. Thanks for bringing it up.
They would bid on something if they weren't going to find a way to make more than what the original contract was for because in the bid wars they often lower the bids so much they have to recoup somehow. In this case they probably did not build the recoup method into the contract and did it illegally.
At least that's my theory.