One piece of the puzzle is that programming education is not very effective. Most computer science programs teach you almost nothing about software engineering, partially because many CS profs know almost nothing about it.
I've been a working programmer [or software engineer if I've just met you at a party and I think you're cute] for long enough to work on >5 multi-billion dollar projects. Of these, one was delivered on time and according to spec. That company went down the toilet within four years.
I've read fairly widely in the software engineering literature starting with Weinberg and Brooks in university, and chasing my share of papers, 'blogs and web pages since then. One of the most positive things that has happened to the field is the recent discovery that the only thing at the end of the methodology rainbow would appear to be consultants [wearing suits with slits for the dorsal fins [h.t. Mr Stross]] and stacks of unread Learn Florble-Oriented Development in 21 days.
I know I can build complex systems. I know I can sometimes build simple systems [much much harder.] I don't know how to reliably schedule a project to build a new thing. I think anyone who thinks they do know this is delusional or lying.
And in the end, I think it probably doesn't matter. The truth of the software engineering crisis, is that it never existed. And it's possible that software engineering never existed either. We program. We do it because we like it and it pays. The folks who do it only because it pays usually end up in management fucking things up by assuming they can motivate people always and only with money.
[oh, on a tangent I've also been a manager. I wasn't all that good at it. The team I managed delivered good stuff, some of it on schedule, but the company still went belly-up. And I learned that I don't enjoy management, and would rather have less money and more fun - take from that what you will]
I can't really imagine that. Are those ever executed by a single company all on its own? Are they software-only? Do they exist outside NASA?
That said, I also suspect he may have been exaggerating for effect :-)
Edit: Google and Amazon likely don't, actually.
But I'd say the iPhone was easily a multi-billion dollar project.
Even if we stick to just the development side, I would be very surprised if Google has not spent > 250 million a year on Chrome development over the last 4 years.
The marketing bit is a bit special since they don't actually spend any money by showcasing Chrome on google.com.
Edit: They do spend lots of money to get it pre-installed on new PCs, though.
Consider Google's opportunity cost for advertising on the "internet's home page". I've read estimates of $2B.
For a while, Google was also paying Adobe to bundle Chrome with Flash (in addition to bundling Flash with Chrome).
As far as people on Chrome go, you're right that back in 2008 they likely did not have 1000+ people on it. The best data I can find for mid-2008 on the Internet says about 50 in mid-2008.
Looking for newer data, according to the last several updates on http://peter.sh/ at this point there are about 100 daily WebKit commits (not all by Google, obviously) and about 120 daily Chromium commits. That's not counting the parts of Chrome that aren't WebKit or Chromium. What fraction of the WebKit commits are from Google is an interesting question; as of early 2010 according to http://neugierig.org/software/chromium/notes/2010/02/webkit-... it was around 40-45% and growing rapidly.
Now going from "number of commits" to "number of engineers" is hard. For comparison, as of today according to http://oduinn.com/blog/2012/09/04/infrastructure-load-for-au... Mozilla has about 200 commits per day, but about half of those are to the automated testing repository, not to the main code. So that's about 100 commits per day. Mozilla right now has around 700 employees, I believe. So if you assume similar commit patterns (which is an assumption, granted), Google certainly has 1000+ people on Chrome now.
But I'm also not sure you need 1000+ people to hit $250 million/year in spending. In addition to Silicon Valley compensation packages, there's also a lot of test infrastructure and whatnot as part of every browser project which costs nontrivial amounts of money to create and maintain.
Again for comparison, since there's data available there, Mozilla spent about $63 million on "software development" in 2010 ($85 million if you include stuff like office space, HR, marketing, etc) according to http://www.mozilla.org/foundation/documents/mf-2010-audited-... and had about 200 engineers at the time, I believe... and Mozilla does not give out huge stock bonuses, like Google has done with Chrome.
Connecting for Health (and all the rest) are the programmes that persuaded me that governments should just define a standard, and then let providers build software and clinicians buy software that conform to the standard.
There are lots of things wrong with email, but SMTP was pretty robust across a range of platforms for servers and clients for many years.
The project developed a new CPU architecture, several variant implementations and ported several operating systems to that architecture. Amongst the more impressive feats I saw during that project:
- a compiler from assembler of architecture A to assembler of architecture B. This was necessary because we had a lot of code [~10^6 lines] in architecture A assembler - several other compilers that handled code written for A, with no thought to portability, and produced asm for B. - a modular emulation environment. Initially it ran on a larger example of A that had a programmable microcode feature. Later it was ported to other examples of A. Some of my colleagues were in the room when the ported OS first booted on a machine of type A with the funky microcode. As I recall, it took most of a day to reach the banner. - my own part was very modest, but I had fun. Amongst other things, I ported a disk checking utility [think fsck] from A to B. That was easy. To test it I had to create bad blocks for it to find. So I got a SCSI 1.44 floppy drive [!], and a box of new disks. Then I formatted and scratched each disk with a paperclip, and did a disk copy.
So, that was lots of bucks, hundreds of software engineers, and some number of hardware people. I don't know how many of the latter were involved. But remember that nearly everything was custom; CPUs obviously, boards, buses [though that was fixed eventually], bent metal, power supplies. The lot.
That was one fairly large company.
I've done other large projects that had many cooperating companies, usually with one or two large partners and a number of small outfits. Those involved similar sums, but a large part of that was in the equipment deployment, land purchase and so on. Not so much fun. I did once crash the wireless data network of New Zealand because a radio [or possibly an antenna] came loose on a mast and flapped around in the wind. Every time it swept through 120 degrees, all the effected clients channel hopped, provoking an exciting synchronisation bug in my multithreaded code.
Oh, and I got paid for all of it. Not as much as I would if I'd become a banker [though it's presumptuous to assume I _could_ become a banker]. I'd quite like the money, but then again I have more fun that the bankers I know. I've built software that was used [and is used] by fairly large numbers of people [if you're reading this, then you're one of them]. This is satisfying. Even it I can't call it 'engineering' with a clear conscience. If computer programming hadn't been invented, I don't know what would have become of me. There aren't that many other occupations for people with my particular blend of what I'll call talents. If you fall into that category too, then get your laughs while you can. You're a lucky person right now.
[and since it behoves me to actually answer the question, I've never done any work for a government or defence contractor. I've no particular objection, it just hasn't come along]
Execution: Generally no. There are sub-contractors, out-sourcing, COTS, etc. And there may be a program management team of companies.
Software only: The three I've worked on, no. Hardware, software, "mechanisms".
Outside of NASA: Yes, although the US government probably has the lock on the sheer number of such programs. Chip-makers, with fab plants, are one sector. Vehicle manufacturers (planes, trains and automobiles) are another. Construction--especially massive high-rises--is probably another.
Enjoying the role is another thing though.
I also echo the concerns below about the "multi-billion" comment. Did somebody just mention sometime that it's that expensive? Because that's a remarkably large amount of money for business. Government? Not so much. But for a business, well, there are very few billion-dollar businesses and a scarce handfull that are spending a billion+ on any single project. My fiance is a PM on the largest electronic medical records rollout in the nation, which includes visits to over a thousand clinics around the country to install hardware, convert paper records and train people, as well as all the infrastructure, and it's reportedly going to bust through the original budget but still clock-in around $100 Million dollars.
As for methodologies, perhaps getting everyone to actually use one is the biggest problem; that's been my experience. I have myself found about 4 things that seem to work:
The project should have a specifications document of one or two pages, no matter how complex. This is to try to get everyone on the same page, as it were, it has goals and requirements, must do N transactions/second, must run on X, etc. As much what and as little how as possible.
Further project documentation efforts must factor in the Don't Repeat Yourself (DRY) principle, i.e. the code is the only truth, and attempts to duplicate it are going to get out of sync unless you work really hard at avoiding that.
From OOSE/RUP: do short ("several week", 2-4) iterations and get your complete software and systems stack into a "Hello, world!" end-to-end state very early in them.
Otherwise, yeah, I've read a lot about software engineering beginning within the first month or two I started programming in the fall of '77 (sic) and while I think there is such a thing, it's indeed not really engineering as we otherwise know it. There's something there, but you learn it as much by doing as by more formal methods (classes, reading books) and there's no substitute for a serious amount of experience.
It's just that there's not that much to learn that is universally applicable, comparatively. The field is young, and then computers are easier to make sense of than most things, because we made them that way. Doctors need to deal with human bodies which aren't designed for human understanding and tinkering. Lawyers need to deal with documents which aren't designed for easy understanding, either - they're designed to prevent hostile, malicious parties from screwing each other. We simply have it easier than most skilled professionals.
And mostly because computer science is not software engineering. If you want to learn software engineering, find a degree or diploma program in software engineering.
Being frustrated that your computer science degree did not teach you about software engineering is like being frustrated that your civil engineering degree did not instruct you how to physically build a bridge, or that your mathematics degree did not teach you how to corporate accounting.