Hacker News Reader

Comment by skrebbel | Hacker News Reader

A multi-billion dollar project.

I can't really imagine that. Are those ever executed by a single company all on its own? Are they software-only? Do they exist outside NASA?

View on HN

The first project; the one that finished nearly on time, ran for three or four years. I joined half way through as a crispy graduate. It came in within a couple of weeks of schedule. It seemed very well managed for the whole of time on it. The project manager left that company, and went to work for Apple. When I googled him just now, all the top hits where at Microsoft. Perhaps he knows how to do software engineering. Or perhaps he got lucky that time. I don't know which.

The project developed a new CPU architecture, several variant implementations and ported several operating systems to that architecture. Amongst the more impressive feats I saw during that project:

- a compiler from assembler of architecture A to assembler of architecture B. This was necessary because we had a lot of code [~10^6 lines] in architecture A assembler - several other compilers that handled code written for A, with no thought to portability, and produced asm for B. - a modular emulation environment. Initially it ran on a larger example of A that had a programmable microcode feature. Later it was ported to other examples of A. Some of my colleagues were in the room when the ported OS first booted on a machine of type A with the funky microcode. As I recall, it took most of a day to reach the banner. - my own part was very modest, but I had fun. Amongst other things, I ported a disk checking utility [think fsck] from A to B. That was easy. To test it I had to create bad blocks for it to find. So I got a SCSI 1.44 floppy drive [!], and a box of new disks. Then I formatted and scratched each disk with a paperclip, and did a disk copy.

So, that was lots of bucks, hundreds of software engineers, and some number of hardware people. I don't know how many of the latter were involved. But remember that nearly everything was custom; CPUs obviously, boards, buses [though that was fixed eventually], bent metal, power supplies. The lot.

That was one fairly large company.

I've done other large projects that had many cooperating companies, usually with one or two large partners and a number of small outfits. Those involved similar sums, but a large part of that was in the equipment deployment, land purchase and so on. Not so much fun. I did once crash the wireless data network of New Zealand because a radio [or possibly an antenna] came loose on a mast and flapped around in the wind. Every time it swept through 120 degrees, all the effected clients channel hopped, provoking an exciting synchronisation bug in my multithreaded code.

Oh, and I got paid for all of it. Not as much as I would if I'd become a banker [though it's presumptuous to assume I _could_ become a banker]. I'd quite like the money, but then again I have more fun that the bankers I know. I've built software that was used [and is used] by fairly large numbers of people [if you're reading this, then you're one of them]. This is satisfying. Even it I can't call it 'engineering' with a clear conscience. If computer programming hadn't been invented, I don't know what would have become of me. There aren't that many other occupations for people with my particular blend of what I'll call talents. If you fall into that category too, then get your laughs while you can. You're a lucky person right now.

[and since it behoves me to actually answer the question, I've never done any work for a government or defence contractor. I've no particular objection, it just hasn't come along]

Thanks for the write up! Makes me want to invite you for a beer, just to hear more about it.

Probably some medium-sized SAP implementation.

Or a Peoplesoft one Or Oracle or anything government or ..

I think timescale may be applicable here. Some key programs for large companies will spend several million a year for several years - particularly if you are taking your Mainframe applications and trying to restructure them to have more easily changed business rules and a new infrastructure.

That said, I also suspect he may have been exaggerating for effect :-)

I have a friend who works for Northrop Grumman, and he just completed a $500 million project he described as "relatively small".

You are kind of right that this type of projects go for years, but other than that, billion+ projects are no exaggeration..

The NHS (Britain's state healthcare provider) famously had a project with a budget of £12.7bn (about $20bn). It wasn't entirely software although that was the main part. It famously fell apart, but billion-sized parts were handed off to companies like Accenture, British Telecom and Fujitsu: http://www.guardian.co.uk/business/2010/mar/21/nhs-national-...

It's horrific. Some bits work and are implemented, other bits don't and wont ever work.

Connecting for Health (and all the rest) are the programmes that persuaded me that governments should just define a standard, and then let providers build software and clinicians buy software that conform to the standard.

There are lots of things wrong with email, but SMTP was pretty robust across a range of platforms for servers and clients for many years.

Answers to your questions.

Execution: Generally no. There are sub-contractors, out-sourcing, COTS, etc. And there may be a program management team of companies.

Software only: The three I've worked on, no. Hardware, software, "mechanisms".

Outside of NASA: Yes, although the US government probably has the lock on the sheer number of such programs. Chip-makers, with fab plants, are one sector. Vehicle manufacturers (planes, trains and automobiles) are another. Construction--especially massive high-rises--is probably another.

Microsoft and Apple both have multi-billion dollar projects.

Edit: Google and Amazon likely don't, actually.

Microsoft yes (example: Windows Vista). The others... no. And that is a good thing, for them.

You're right about Google and Amazon, early posting on my part.

But I'd say the iPhone was easily a multi-billion dollar project.

I seem to recall iPhone 1.0 being called a $100 million dollar project in some Wired article. They were several hundred people (not thousands), working together for about 2-3 years. That seems sensible, and the numbers seem to add up.

Apple's strategy is to assign fewer people than you would expect when building a new product to ensure focus / reduce feature creep. Staffing on stable products are in line with what you would expect.

For example, I heard (from an ex-Apple co-worker) that the entire iTunes.app team was about 25 engineers. In contrast, Adobe's Flash plugin team had about 50 engineers (not including QA).

Google sure has multi-billion dollar projects. Chrome is an example. They're spending that much just _marketing_ the thing every year.

Even if we stick to just the development side, I would be very surprised if Google has not spent > 250 million a year on Chrome development over the last 4 years.

You think Google have had 1000+ people on Chrome for the past four years?

The marketing bit is a bit special since they don't actually spend any money by showcasing Chrome on google.com.

Edit: They do spend lots of money to get it pre-installed on new PCs, though.

1000 engineers? Probably not, but likely many hundreds plus all the organizational support people like designers, product marketers, product managers, people managers, and admins.

Consider Google's opportunity cost for advertising on the "internet's home page". I've read estimates of $2B.

For a while, Google was also paying Adobe to bundle Chrome with Flash (in addition to bundling Flash with Chrome).

Even ignoring the google.com thing, estimates are they're spending at least $500 million a year on Chrome marketing for ads on other sites, various bundling deals (including pre-installs on new PCs and auto-installs with other software), etc.

As far as people on Chrome go, you're right that back in 2008 they likely did not have 1000+ people on it. The best data I can find for mid-2008 on the Internet says about 50 in mid-2008.

Looking for newer data, according to the last several updates on http://peter.sh/ at this point there are about 100 daily WebKit commits (not all by Google, obviously) and about 120 daily Chromium commits. That's not counting the parts of Chrome that aren't WebKit or Chromium. What fraction of the WebKit commits are from Google is an interesting question; as of early 2010 according to http://neugierig.org/software/chromium/notes/2010/02/webkit-... it was around 40-45% and growing rapidly.

Now going from "number of commits" to "number of engineers" is hard. For comparison, as of today according to http://oduinn.com/blog/2012/09/04/infrastructure-load-for-au... Mozilla has about 200 commits per day, but about half of those are to the automated testing repository, not to the main code. So that's about 100 commits per day. Mozilla right now has around 700 employees, I believe. So if you assume similar commit patterns (which is an assumption, granted), Google certainly has 1000+ people on Chrome now.

But I'm also not sure you need 1000+ people to hit $250 million/year in spending. In addition to Silicon Valley compensation packages, there's also a lot of test infrastructure and whatnot as part of every browser project which costs nontrivial amounts of money to create and maintain.

Again for comparison, since there's data available there, Mozilla spent about $63 million on "software development" in 2010 ($85 million if you include stuff like office space, HR, marketing, etc) according to http://www.mozilla.org/foundation/documents/mf-2010-audited-... and had about 200 engineers at the time, I believe... and Mozilla does not give out huge stock bonuses, like Google has done with Chrome.

Reply on news.ycombinator.com