Limits of Programmer Productivity: A lesson from Fred Brooks
pboyd.io
pboyd.io
Your programming workflow was:
1) write your complete program on paper
2) grab a box of fresh punch cards, find an empty and functioning card punch and type out your complete program onto cards.
3) very carefully pack them and walk them over to a card reader. Yes. I saw on more than one occasion grown men brought to tears because they dropped a box of cards scattering them across the floor and they would have to spend many hours sorting them.
4) stand in the queue at the reader waiting for your turn to read in the cards. In some places you did this yourself. In others you had to hand your cards to an operator that controlled the reader.
5) if you were in the main programming room of the datacenter, you would now walk over to the queue at the line printer awaiting the output of your program. Each output on green/gray continuous linked A3 paper started with a full listing of your code as typed. If there were errors in your code this would be followed by a huge pile of full code repeats indicating were things went wrong. I still think the enormous stack of pages printed in those cases was mainly to shame the programmer as they had to stand there waiting for the relentless ongoing machine gun like soundings of the line printer anoucing your failiure to the chuckles head shakes of the others standing in line. However, if you were at one of the satelite programming rooms spread around the campusses, , now was the time to go home as your output print would be delivered by van in the following batch typically the next day when you would go and pick up your output from all of them lying on a set of tables, hoping yours was not one of the bigger stacks indicating an error had been found.
My dad was just a mechanical engineer, and just crunched numbers with punch cards, but even he has stories about worrying about dropping punch cards.
I realize I may be a bit late with this suggestion, but why not number the cards, like the pages of a book?
However that means that it was possible to sort them, not that it was possible to sort them (manually) in a short time.
Since a card was a program text line, there were frequently many thousand cards in a program and the smallest programs had a few hundreds.
Fresh from my algorithms course, I organized a parallel meat space merge sort. And it went pretty quickly.
Everyone started with very small random stacks to order, and then merging the pre sorted stacks was relatively quick mindless work.
I like it
At least that's what I heard from the older professors at my university.
Given how often you hear about such disasters, I wonder why nobody thought of numbering the cards?
IF statements looked like this:
IF (Y+Z) 100,200,205
My first programs were punched with the IBM 026 keypunch machine. This was a machine not really intended to punch program decks. It was for the earlier use of the cards to hold data that would be tabulated, sorted, or collated by IBM tabulation machinery. The later 029 model keypunch was a huge improvement for programming, it even had keys that would punch a parenthesis!
There were special file cabinets to store long stacks of cards in tray like drawers. You could pull out a drawer holding maybe 1000 cards and carry/drag it over to be submitted for a run. Smaller programs or data sets were carried around in shallow boxes.
I don’t ever remember having a dropped card box disaster, but I would mark my card decks with thick diagonal lines so that out of place cards could be easily spotted.
Even a simple, hundred line program might take 30 minutes from the time you handed it over to the computer operators to the time you saw any output (which came out on fan fold wide paper print outs). The operators lived in a big bright room with a glass wall housing the big computer with blinking lights and wore sweaters and could only receive your submitted programs through a small window. The operators literally looked down on us lowly programmers. (The computation centers had elevated flooring about a foot high to hide all the power and cooling lines.)
Unless you were special, you weren’t allowed in the secured room where the corporate computer resided. The corporations were very proud of their giant electronic brain—that’s why they put it in the big, bright, cold room behind glass so we could contemplate its power. The corporation might have a second computer, but it would be in a different city (in case of an earthquake or a Soviet nuclear bomb).
Because of the slow turn around time, a syntax error would require 30 minutes to discover. Then fixing it would require fumbling with cards to find the error and finding a free keypunch, more blank cards and then punching and fumbling around some more to refactor the code. Thirty minutes later . . . And you find your second syntax error.
Turn around time varied greatly, some times I was allowed only one overnight run every couple of days so I wouldn’t see my output until morning. This gave me lots of time to desk check my program. Sometimes I would even find my syntax error before the computer did.
Back in the bronze age, programmers didn’t have text editors, Github, or even a file system to store our source on, we just had drawers to hold our cards.
This link has pictures of the IBM 026 keypunch (used by the Vikings).
You know what? I like it.
> IF (Y+Z) 100,200,205
> This evaluates X+Y
Found your bug! Please resubmit for evaluation in 24 hours.
We actually got our printouts more or less right away but computer time was very limited at least for non-CS undergrads basically taking a programming course.
I've deployed 35,000+ lines of code to prod in 2023 with this flow. I've only had 2 small bugs.
This is by far the most efficient setup I've ever used.
(I guess I'm being facetious because I think everyone should read his book of essays. The hardware platforms have changed dramatically since the 1960s, but the wetware hasn't changed a bit)
In theory if your process is perfectly optimized and everyone is doing the correct things all the tike and the goals are well defined then this is true. The biggest bottleneck I see in most projects are that there little to no definition in what needs to be done. It's not a handholding task, it's a lack of clear or continuously fluctuating goals. That ultimately ends to people slowing down.
Business leadership wants to hand the goal of "make money" to the engineering team but IMHO that's an unrealistic expectation and shows poor, incompetent, and or lazy leadership. Someone needs to find the demand signal and be decent at predicting future demand to direct the team towards what needs to be created. At some point, if you think you can hand off lofty lazy goals to an engineering team like this then your role becomes questionable because you have a hybrid engineering/entrepreneurial team who could basically work without your involvement.
Nits aside use the same style and have set a time budget of 50ms for myself. I have even setup nvim to save on every keystroke (one also learns how to not write infinite loops when programming in this style).
This style of programming is transformative because it's like having a persistent repl. It becomes feasible to test every edge case with print debugging as you go. This workflow is also why I have little interest in any language that primarily targets llvm, which will easily blow my time budget in code gen.
/s
I've had this period where I worked with another contractor and we just blended in like one. We spoke twice a week and we got so much done because we both knew what needed to be done and got the ability to do it.
The key is that *the sum of the people should be worth more than the individual output of each*. The larger the team, the harder this is to achieve.
2. This classic from xkcd really makes a difference https://xkcd.com/1205/ - Automate, eliminate, speed up. The sum adds up to massive savings.
3. Minimise cross-teams interruption. No sync up between teams, or having repeat meetings about the same thing; have clear meeting agendas. Divide the work in asynchronous output. For this you need good documentation and standards. Don't let other teams set your standards.
4. Have clear priorities. I've seen teams waste huge amount of time because priorities weren't clear. Flat-orgs are a thing, sure, but you better be able to justify how you spend your time. Consider every hour as a billable, not as a lifestyle.
5. Let people do their job. Because they're quiet doesn't mean they're not working. You might not be busy but they might be. Treat silence as deep work, not stillness.
Many personal hours are wasted because we're waiting on other people. Many team hours are wasted because priorities aren't clear or not well allocated. Finally, we're all slaves to slow systems, builds, deploys: be ruthless about those.
These are the main culprit of waste (imho!).
1. Know your tools inside out. If you're a Vim user or not (I'm not), make sure you do most things with shortcuts as much you can. Take notice of every time you use the mouse and consider whether that could be keyboard driven. You'll first save seconds, which turn into minutes, then hours.
1b. Same for the terminal tools. My 'profile' is shock full of aliases and functions that make my life easier to talk to containers, other tools, etc. It almost becomes a library of power tools.
2. Effective navigation through code, forward, backwards, definitions, declarations. These should become second nature and effortless to save a ton of time!
3. Use every power tools available, whether it's static analysis tools like Rust Clippy, or language servers (LSP) in VSCode, Neovim or elsewhere; or chatGPT, make sure you make technology work for you, as much as possible.
4. Hook up to Dash or DevDocs to reduce your time clicking about Google trying to solve a problem. Knowing everything is impossible, knowing where to find the information quickly is a life saver.
5. Sometimes you'll find a really hairy bug, document it for your future self. Whether you blog it up at night or simply keep it in your (searchable) notes, it can save you 20 mins down the line, it's worth it. Sometimes it's a recipe list of "how do I do that with X" systems that can't be automated. If you can go home and forget everything, you're doing it right.
6. Turn off notifications etc. as much as possible. This is well known... It's perfectly fine to browse HN in my mind as it's a net addition to my life, but be conscious about time spent on socials and more importantly knowing when to turn them off.
And some automations have similar steps so you can borrow from these you already have
And on the other dimension, of "how long it is to write a program", well, it's a simple matter of precisely predicting the effort for a development task. We all know developers are great on this.
Overall, this is a fun comic, but people shouldn't read anything else from it.
I mean I write system tests of whole applications using pytest where the embedded software is built using emulated HW.
This allows me to test changes locally before I test on embedded hardware. This is a lot quicker than flashing firmware and testing on real HW.
Of course this is not usable for everything. Having a HAL layer helps.
This simulator would be run on a phone to validate UI and functionalities prior to hardware implementation.
Much easier to iterate over mobile software than hardware and perhaps card firmware, I guess.
I found that the code I write has far less Code Mass relative to its functionality. This means fewer developers are needed relative to a product's functionality as well, which contributes to less communication. It's a huge cascading effect that is invisible unless witnessed within an organization as a whole.
Nix solves this problem.
(While introducing another one- learning Nix- but I digress)
That seems a bit extreme. On the other hand, where a complete build and tests take excessive amount of time, running everything by everyone would be similarly bad.
Maybe the optimum is in between. Quick checks with a reasonable chance to catch frequent problems by everyone, long-running rarely failing checks centralized.
Increasing the "number of shots" per unit of time doesn't buy you anything if the shots you take yield proportionally lower output. For example, in the days of punchcards and batch processing, if you were unsure about the syntax of one line of code, you wouldn't have just run the compiler, as it meant wasting an entire cycle. So the productive output of one run of the compiler back then would have been much higher than one run of the compiler nowadays.
Also, hitting a wait-state on any one task doesn't necessarily block the programmer as a whole, and doesn't necessarily decrease that programmer's throughput. You could imagine a scenario where a programmer is on ten projects. Whenever they get blocked on project A, chances are high that among their other nine projects, there is some project B that became unblocked while they were working on A. ...so, in theory, the programmer would never themselves be blocked, they would always have something productive that they could do, and the throughput of an entire team or company could remain unaffected.
The reason why there's negative productivity impact to organizing work in such a way has to do with "context switching overhead" when a programmer needs to context switch between different tasks/projects, or, if you will, a kind of "cost of retooling" a programmer's brain to focus on a different task.
It's also a productivity trap for psychological reasons. On one hand, you have the positive psychological impact of "flow state" which requires immediate feedback. On the other hand, you have a situation where, every time you have to wait for something, it's an invitation to focus on a distraction instead of something productive.
And then, last but not least, the amount of time a project takes to finish (talking latency now, rather than throughput) is a function of how densely you can schedule the tasks that are on its critical path, and if the programmers are actually in a position of having to context-switch a lot to stay productive, then that would introduce a lot of wait cycles into the critical path, unless it were counteracted by deliberate planning of the kind which is extremely hard to do, unpractical, and would come with its own overhead.
I think it's things like that which Brooks must have intuitively grasped when he made that statement.
I think it's also important to realize that Brooks' project had to succeed, or IBM would go out of business. The company was nearly done for, and Brooks tried to write down what he learned because those lessons saved IBM.