Moving Past the Scaling Myth
michaelfeathers.silvrback.com
michaelfeathers.silvrback.com
If you pay attention, the best Agile systems focus on improving the skills of developers instead of forcing people to follow the 'steps' or a formula.
Of course a design that works at 1x doesn't always work a 10000x, but why shouldn't a design that works a 10000x not also work at 1x?
At least for technical scalability the answer almost always lies in overhead. Yes, while the system scales perfectly linearly at O(n), the hidden constant is often neglected, which means at small values of n the solution becomes unfeasible. So while Cassandra might for one of my use cases be the perfect candidate due to its feature set and linear scalability it becomes unfeasible at a small scale because I have to run at least 3 nodes with high overhead (because Java, sigh) + an additional 3 nodes with Zookeeper with high overhead (because Exhibitor + Curator, all in Java).
/rant, I don't like bashing on Java but overhead is definitely one of its weak spots, and here is where it comes into play.
I can understand why such systems are built the way they are, but I still find it sad, since it breaks the expectations of open source software, and creates a chasm between startups at the beginning and end of a big scaling phase. It more or less forces you as an early stage startup to use an inferior design which you know won't scale up beyond a certain point, even though know about a superior design that theoretically scales down to your current requirements.
If you sit down and try to design a system to support 100000x your current workload, you are likely to end up with an over-engineered and very expensive mess, then by the time you reach 10x you realize you should have done it differently because in practice the bottlenecks aren't where you thought they'd be. This strategy will bankrupt you before you get to even 10x.
This is how you tell who has really scaled systems, from those who only wish they had that problem.
You have alternatives, and it doesn't really have to mean a whole lot of extra work until you get to "crazy" levels. No one wants a StrategyFactoryFactoryVisitorImpl, but on the other end of the spectrum you can get a ton of mileage out of a few Facades in the right place.
Scalability is an optimization topic. But just like the premature saying everyone knows, the follow up that is less acknowledged is maybe the more important part (IMO).
The problem I see is that people aren't really willing to be honest about the cultural transition that happens, so we also can't be honest about the process transition.
I think Agile-ish approaches work very well in startups, because the structure is pretty flat, and the goals are shared. But as companies grow, they tend to become what I think of business feudalism: hierarchical, control-oriented, territorial. For that, it makes sense you need different processes. And I think large company Agile is in effect Waterfall with a faster cadence, so you get that different process. But nobody will admit it. "We're doing Agile," they say, with too-bright eyes and gritted teeth.
What I wonder is: what if instead of killing the peer culture and the human-centered process as we scaled, we kept them?
I think that the big challenge is to recognize that hierarchical structure isn't good or bad, it's something that most natural systems do in response to scaling. If we understand it as a design parameter we can make it humane.
When you talk about hierarchy as natural, I'm not sure we're talking about the same thing. I mean hierarchy as a system of power/control relationships, with each unit having an actor in charge and totally containing sub-units. E.g., the classic army chain of command. I can't think of anything natural that has that. Flocks and troops have leaders, but I think humanity is unique in its extension of social dominance to many-layered nested structures controlling millions of individuals. So I'd call it essentially artificial.
I agree that we can apply a hierarchical analytical model to many natural systems, but I think there are plenty of other models we could apply, just ones that aren't as easy for current human brains. E.g., we could talk about fractal, self-similar, or recursive processes; those have no connotation of primate dominance. Or note the shift away from "great chain of being" analyses of nature toward ecological and non-teleological analytical models. When all you have is social primates, everything looks like a hierarchy.
I would certainly like to believe that we can make hierarchies humane. But given that a fundamental characteristic is individual power over others that increases as you go up the chain, and given human fallibility, cognitive limits, and human nature, it seems to me they have to become essentially inhumane at some size. And in practice I certainly see a correlation between hierarchy size and inhumanity.
Correlation isn't proof, of possibility, of course. Just natural tendency. Are there large hierarchies you think violate that correlation, where they have gotten more humane with scale?
Ever worked in a chain restaurant? Camaraderie among coworkers at a site under a manager (so hierarchy) and not as much peer connection across sites.
I'd argue that you are using a narrow definition of hierarchy that is centered on ideas about power rather than dynamics in emergent systems. The arterial structure of a leaf is hierarchical. So are airport connection patterns. Whenever you have preferential attachment or costs associated with material or information transfer there's a force toward hierarchy, designed or not. Systemic effect. No primates needed.
The interesting bit to me is that if we considered something non-sentient, like a network router that controls flow over a set of nodes, it would be easy to anthropomorphize it purely based on its behavior and see it as exerting dominance when it makes its routing decisions if they don't align with what we want at that moment. Although there can be abusive hierarchy, there's something deep to be considered in that scenario. We might be inclined to see all hierarchy as bad because that sort of conflict of interest can happen, willed or not. That's why, where there are power relationships in employment or even representative democracy, systemic integrity and human values depend upon gaining and maintaining consent.
I have worked in a chain restaurant. And that's one of the experiences that convinces me of the essential inhumanity of deep hierarchy. There was some coworker cameraderie, sure. But I don't think there was anything human-centered about the place. Workers were replaceable, disposable units. Much of the worker bonding was despite (or against) the company and the managers, not because of it. And restaurants in some ways have it easier, in that the purpose of the work is present and visible. For many organizations, that's much less true.
As to definition of hierarchy, the last half of the word literally means "rule" or "government", like monarchy or oligarchy, so I'm comfortable with my definition. I agree one can use it as a metaphor to understand natural systems, but it's only a metaphor. And as with the Great Chain of Being, I think it's a dangerously easy one to over-apply.
There are surely phenomena in emergent systems that relate and are worth studying, but conflating them with primate dominance structures obscures more for me than it reveals. To the extent that nature tells us things, though, it can't tell us what ought to exist. That's our job.
I totally agree that consent is vital for humane organizations. But I think that's in direct opposition to hierarchy, which is about top-down control. American democracy, for example, is basically non-hierarchical. I'm a participant in and subject to at least 9 quasi-independent governments (city, county, state, federal, regional transit district, school district, community college district, air quality management district, municipal utility district), plus various splits (executive, legislative, judicial). None of these controls any of the others. And none of them controls me, either, except in certain deeply constrained circumstances.
That stands in contrast to me to the essentially feudal structures of scaling American businesses. The main rule there is "my way or the highway" for all superiors. In theory, employees can always switch jobs to another company. But that's always much easier on the employer than the employee, so I think that does very little to act as a practical check. Employees and teams can try to carve out niches, but I've always seen that happen at the sufferance of (or with the neglect of) management, never because the teams had much real authority. And the more levels they have above them, the larger the set of people who can ruin things for them without consequence or even awareness.
The only exception I personally see much is at very mission-driven organizations. There the stated purpose really seems to have some teeth; people are willing to leave if they aren't doing good. In fact, right now, one team at Code for America has just demanded the resources to improve their code quality so that they can be more effective at serving their audience. And I just saw them excitedly share your 2002 paper. So I believe it's possible!
I guess what I'm arguing for here is to see what happens when we put human-centered systems first and scaling second. It may be that there are fundamental limits, in which case we'll face some hard choices between economy of scale and treating humans like humans. But I've never even seen a place try. It's always, "Oh, we're getting big, better hire an earl of mobile development and put him under the duke of engineering." Maybe the bankers overthrow the CEO boy-king, maybe they don't. But I've never seen anybody stop to question the dominant paradigm.
Realistically, they have options in both spheres, but they only feel one oppressively and they tend to assume that everyone else does too. Where power is exercised at work they are okay with it - they recognize it as part of the system and don't feel like cogs.
I wish you the best in finding new glasses to see the world of work through, just as I wish the same for people I know who are fighting governmental oversteps. The key thing is - it's not all bad. There is badness out there but structure is not determinative. Look for happy people in any system. You'll find them. What we look for in life does affect what we see, and what we create. Best of luck to you.
When administering computers we talk about treating them as "cattle, not pets" to increase our ability to manage large systems. Using the same metaphor with people would be offensive but the policy-making impulse is similar.
To some extent it even arises for valuing fairness. We make rules to try to treat people consistently and fairly, forgetting that people are a mess of special cases.
Small companies value consistency too (using the same tools and procedures) but it's just easier and less likely to backfire with fewer people.
In which case, I'd like to see what happened if we favored humanity over scale.
So, I don't think we can take it as far as author is suggesting where it's like classical vs quantum physics. Maybe at ASIC vs software level. Even those were partly joined with synthesis & coprocessing tools. I just don't see it as every high-level description I read about things is based on similar principles at each layer of the stack. Given similar constraints or goals, you would use similar strategy. There's certainly divergence or outliers but more repetition of patterns than anything.
The only myths are that technology/fads X, Y, and Z should've been widespread adopted over the ones (or enhancements of them) that consistently worked. The author is seeing results of people building on stuff that came with assumptions that don't match new problem. Or people straight-up ignoring root cause in their solutions or methods. Common problems.
What about starting off with microservices (because tools like k8s make the otherwise insane management overhead more tractable)? Could that be a possible solution?
The problem of scaling state, databases need to handle anyway. Ideally the same database just keeps on working as you go up. Or atleast the protocol is the same, and you switch out the implementation from a single instance to a clustered version.
So, every process around being able to deliver that by just applying that process creates a false hope. Or it is not meant for this purpose but the "buyers" aren't aware of it and have other expectations.
I'd like to give you a simplistic answer, like "All you need, kid, is a small team! For anything!"
Slogans like that are true, but yet they are terribly misleading, because 1) many organizations are already terribly overstaffed, and 2) it doesn't really help to tell teams "What do I do right now?"
So here's as simple as I can make it:
Good organizations will do whatever is necessary to make things that people want, even if that means instead of programming, the programmers sit on the phone and do some manual chore for people as they call. Before you code anything, you have to be hip-to-hip with some real person that you're providing value to.
But as soon as you have those five folks sitting on the phones doing something useful? You gotta immediately automate. Everything. This means you're going to have all freaking kinds of problems as you move from helping ten people a day to helping a million. You have to automate solutions, access, coordination, resource allocation, failovers, and so on -- the list is almost endless (but not quite)
As they grow, poor organizations take a scaling problem and assign it to a person. Somebody does it once, then they're stuck with it for life. Good organizations continue to do it manually, then immediately automate. Somebody does it once, then the team immediately starts "plugging all the holes" and fixing the edge cases so that nobody ever has to manually be responsible for that again.
Growing "sloppy" means you end up helping those million people a day -- but you have hundreds of people on staff. Meetings take time. Coordination takes time. There are a ton of ways to screw up. People tend to blame one another. Growing good means you can be like WhatsApp, servicing a billion users with a tiny team of folks.
If you're already an existing BigCorp and have been around for a while -- odds are that you are living with this sloppy growth pattern. That means you need to start, right now, with identifying all the feedback loops, like release management, and automating them, such as putting in a CI/CD pipeline. Not only that, but there are scores of things just like that. You have a lot of work to do. It might be easier just to start over. In fact, in most cases the wisest thing to do is start over.
Now picture this: you're an Agile team at BigCorp and you've got the fever. Woohoo! Let's release often, make things people want, and help make the world a better place. But looking around, all you see is ten thousand other developers in a huge, creaky machine that's leaking like a sieve. You go to a conference with a thousand other BigCorps, just like yours. Are you going to want to hear about how it's better just to trash things and start over, about the 40 things you need to have automated right now but don't, or how to make your section of 150 programmers work together; how to "scale agile"?
Scaling Agile is an issue because the market says it is an issue.