Focus and Flow: trade-offs in programmer productivity (2021)

I do wonder if group flow and individual flow have more to do with different skill sets, and which path best suits the people you have, as opposed to the idea that group or individual flow is more effective.

Like, I like to whiteboard and mock up a problem in Python to wrap my brain around it. My coworker likes to mock things up in Excel. Usually when we work together, I find myself just waiting to get off the call so I can do my own whiteboarding. What makes sense to him in his Excel file doesn’t hit as much for me.

Whereas it seems like group flow uses some kind of higher level coordination process, whether it’s whatever orchestra “best practices” have been established over centuries, or people chanting together in temples, or whatever.

The question being, while group flow exists, is it simply that it requires people who like engaging in the “group flow activities”? Like we can probably find people who coincidentally like chanting, and those people might make good monks. But will my coworker and I ever work well in a pair programming environment?

I've played in a large band, with over fifty members. I did that for several years. The thing that kept us together was the conductor, not some sort of "group flow".

This is not to say that "group flow" doesn't exist, but I'd want more extraordinary evidence for this claim.

I use mini-pomodoro to build momentum when I can’t get started.

Start with 2-3 minutes “work”, 1 minute break. Keep bumping those up toward 25/5.

Usually by the time I’ve done a few rounds, I don’t want a break and I’ve got my momentum, and I ditch the cycles and just work.

This works surprisingly well for me even at a micro level, like exercise 10 seconds, break for 5 seconds, repeat. Before I know it I’m just exercising uninterrupted for 20 minutes.

I left out one part: clap for yourself

The process is more like:

* Pick something

* Do it

* Clap for yourself

* Repeat

And the key is to make “do it” as small as necessary to keep the cycle going, then expand it once the human reward system takes over.

There are more reasons to do pair programming: It also transfers knowledge of all kind (domain, tools, techniques), saves the time for code review, strengthens relationships and improves team culture

It has always been strange to me that for an industry that claims to be engineering and data focused stops when it comes to our own processes.

One Lazy google search searching for a paper, and not a blog:

https://collaboration.csc.ncsu.edu/laurie/Papers/XPSardinia....

  In 1999, a controlled experiment run by the
  second author at the University of Utah
  investigated the economics of pair programming.
  Advanced undergraduates in a Software
  Engineering course participated in the
  experiment. One third of the class coded class
  projects as they had for years – by themselves.
  
  The rest of the class completed their projects with
  a collaborative partner. The results of how much
  time the students spent on the assignments are
  shown below in Figure 1. After the initial
  adjustment period in the first program (the
  “jelling” assignment), together the pairs only
  spent about 15% more time on the program than
  the individuals [4]. Development costs certainly
  do not double with pair programming!

  Significantly, the resulting code has about 15%
  fewer defects [4]. (These results are statistically
  significant.) Figure 2 shows the postdevelopment 
  test cases the students passed for
  each program – essentially the percentage of the
  instructor’s test cases passed.

  The initial 15% increase in code development
  expense is recovered in the reduction in defects,
  as the following example illustrates...

So the from this paper is the initial work is more costly, anyone that tells you it is not is probably wrong, but not double. But they are producing different work at a different quality level, at a different speed.

Sure there could be problems with the methodology, it might apply to students and not professionals (a classic psyche 101 bias). But it is something, certainly not a scam.

Personal productivity is second to team productivity. Pair programming facilitates discussion, context sharing and alignment. The biggest problems are almost always misalignment and lack of context/shared understanding, not productivity of individual team members.

Pair programming is, first and foremost, a technique for implementing continuous review. The old adage: If code reviews are good, do them constantly and continuously. We pair/mob when appropriate, and we don't do pull requests. If you pair AND do pull requests, it may be worth examining that.

The secondary order effects cannot be ignored either. On our team, pairing is used to:

- Coach more junior developers into being more proficient at their work

- Share knowledge so that norms, standards, and other knowledge can propagate through the team in a more natural (and effective) way than documentation

- Share techniques at the more tactical level

- Further explore the solution space and do better planning and execution (go slow to go fast)

And others.

As an entrepreneur and engineer and programmer having spent 30 years in software without washing out and still accelerating, I'm shocked every time I encounter the well-conditioned presumption that work cells in knowledge work should be decomposed to the single point of absolute irreducibility, which is inevitably the 1-person/1-work-item presumption. It implies the assertion that parallelization _should_ be maximized to the ultimate, which implies that the nature of the work is largely just implementation details and typing out code on a keyboard. But that's not where delays and inefficiencies are in software. And decomposing work cells to the point of total parallelization only exacerbates the factors that do indeed worsen the delays and inefficiencies.

Software development has been so underperforming for so long that we presume that the natural, unavoidable state of software development is what the average team experiences today. That's only true of average teams with average productivity and average per-person yield. It's true in the middle of the productivity/performance bell curve, but not to the right of it.

The 1-person/1-work-item division of labor sub-optimizes the momentum transfer that high-performance teams harness to get the high-performance results that they're after. Doing so is predicated on a number of factors, but principally it requires a vigorous attention to structural design fundamentals, rigorous norms, the ease of provability of changes and additions made to the design and implementation, and a solid grasp of organizational mechanics and process mechanics with a fluency in the effects of batching and queueing in systems-of-work in knowledge work and product development.

In my work, and in the work in our organization, the 1-person/1-work-item system is the exception and collective work is the norm. That's because of breakthrough levels of productivity that can be unlocked at the upper echelons of quality. In fact, at that level, the terms "productivity" and "quality" don't have any meaningful difference.

In the middle of the software development curve, there is already such middling productivity/quality that any particular strategy is unlikely to have much observable effect at all on outcomes. It's not until quality is increased to a point well over the present average that the effects of the numerous individual virtuous systems amplifying each other in the overall production system that the sub-optimization of the 1-person/1-work-item division of labor is obvious.

At that level, management of software development operates like an accounting system that only records gains and disregards losses from the final bottom line. Every iota of spent effort is counted as a gain without regard for whether the result of the work creates loss-free assets or not.

The field of software development has largely lost track of structural design fundamentals, both in the software itself and the design of the processes and procedures wherein the software is built. The vast majority of software developers operating today don't know what a mistake is in software development. They don't even recognize the design mistakes inherent in the tools they choose, and they tend to live in software ghettos where productivity poverty is the norm. They've got good slum survival street smarts, but have never been outside of the slum, so they couldn't orient themselves toward a healthier approach and healthier work life if they wanted to. They don't recognize the mistakes that are painfully obvious when seen through the lens of design fundamentals, so they don't know what to avoid. Software developers at large don't know where the "true north" heading is.

To me, it's this current state of ghettoized software development that's the scam. It's the productivity balance sheet that fails to recognize losses and setbacks and that books "technical debt" as an asset that is the real scam in software work and software work management. But again, the vast majority of software workers and managers have never known any better.

So, yeah... putting advanced, interconnected systems of work in the hands of teams that labor under the conditions that can't benefit of those systems of work is obviously ill-advised. But that doesn't mean that those teams can't induce productivity breakthroughs and realizations by practicing the techniques, even if in an uncoordinated and indefinite way. Though it will look like inefficiency to a manager who also has never had the opportunity to work outside the limits of software poverty.

For a team to be able to go beyond single-person work cell division of labor and parallelization, it'd have an enormous amount of historically-cumulative deficits to overcome. And a team that's operating like this is operating with such relatively-low throughput that they're likely already to behind on their deliverables even before they get started on work that it would be impractical to suggest taking a more deliberate approach. In short, this kind of team is always operating under what are effectively emergency measures. That team's principal management system is Crisis Management. And when every work item is managed as a matter of crisis management, there's no point in hoping for much more than mere survival and coping mechanisms.

I wouldn't say that it's necessarily a "scam" to fail to manage the endless emergencies and crisis intentionally, but it should raise some eyebrows for a team to be so far behind the eight ball to be experimenting with systems of work that aren't supported by the counter-balancing systems of work that interact to create emergent productivity. But then again, if they don't start some time, they'll never really get ahead of the software poverty line.

The real scam in software development is "technical debt". The normalized predatory lending and borrowing that the term "technical debt" is a proxy for is the real racket in software development. Being a "post technical debt" team is something that few developers who started their careers since the term was coined can even imagine. It's like a mythical Lost City of Atlantis to developers who've never known any difference. The technical debt that keeps software works and software workers in technical poverty is the issue that should be hauled out into the cold light of day and exercised like a malevolent demon.

But that can't happen until software developers, designers, managers, and operators invest the time to come to an understanding of what structural design decisions makes software work harder and increases the level of poverty that developers live in, and keeps increasing it until the developers themselves are utterly desensitized to their own unfortunate circumstances.

But if they can make it out of the slum, getting past the grasping at 1-person/1-work-item management is part of it. That person will inevitably realize that it's an entirely arbitrary structure born out of the desperation of a project that embraced its own decline in the very first moments of its life. It's a work system that is an effect of the same conditions that make software work harder to do than necessary. At that point, every developer and every manager will have an intimate grasp of the difference between efficiency and productivity, and will know not to forsake long-term productivity for mere short-term efficiencies, and managers will know better than to manage for utilization rather than continuity.

But it can't be readily done in a team that's already implemented itself into a technical deficit for not having known what to avoid in the first place.

The scam is the low-yield spend of most software efforts. "Cooking the books" is today's software development's stock and trade. In a very real sense, software development is largely operated as negligence at best, and fraud at worst. And a good deal of the negligence is the failing to recognize that the sub-optimizations that we submit ourselves to every day are self-inflicted.

The most heartbreaking thing about the whole sordid software affair is that to solutions to the problems have been well-known for decades in industries that have more maturity than software, but software's industrial immaturity is the very thing that keeps it from seeing the need to make a study of industry and production systems.

Taylorism is the sickness that doomed so many industries that came before software, and it remains the mindset that continues to suppress software work today. And nothing will change until developers start to really, really make an effort to understand the subtle reasons why things are the way they are, and to start working for a life beyond the self-inflicted poverty of the ghetto and the slum. But generational poverty is a much more pernicious and endemic issue in software work than the appearance of the petty crimes and scams that impoverished people give into just to make their lives a little more survivable.

Scam? How so?

I believe that thinking your solo work is good enough is hubris. I'm not a fan of pair programming either, but I don't trust my own work enough to not have someone else review it first.