Management by metrics leads us astray
jakobgreenfeld.com
jakobgreenfeld.com
Also, this reminds me of stories about Soviet manufacturing quotas[1]:
> The Soviet Union was the largest producer of shoes in the world. It was turning out 800 million pairs of shoes a year–twice as many as Italy, three times as many as the United States, four times as many as China. Production amounted to more than three pairs of shoes per year for every Soviet man, woman, and child.
> The problem with shoes, it turned out, was not an absolute shortage. It was a far more subtle malfunction. The comfort, the fit, the design, and the size mix of Soviet shoes were so out of sync with what people needed and wanted that they were willing to stand in line for hours to buy the occasional pair, usually imported, that they liked.
Or even more ridiculous[2]:
> But probabably the most outrageous malfunction of the system was of a factory that fabricated metal products. It had a quota for scrap metal for recycling. The factory was operating more efficiently than expected so it was not generating as much scrap as expected. Since the factory was not meeting its scrap metal quota the higher authorities were going to fine it. So the factor personnel did what to them was rational. They took perfectly good zinc metal sheets and converted them into scrap to fulfill their scrap metal quota.
When looking at large organizations through the lens of information problems, one realizes that Amazon, Google, Walmart et al. all exist on continuum along with the Soviet Union or even the US Government.
[1]https://www.econlib.org/archives/2009/09/soviet_shoes.html
Sometimes he says things that are, well, nonsense (e.g. a lot of his anti-GMO posturing), and other times he's right on the money (e.g. criticizing Black-Scholes long before the '08 crisis). In all cases, he insults his detractors as "bullshitters" when it is only warranted for some.
Internally, corporations function like centrally planned economies.
Well, they try to. But unlike Governments, they don't have access to Departments of economists or planners that can spend all their time looking at the data and deciding the optimal plan. The decisions corporations take is most often based on the whims of its decision makers (senior managers/HOD/VP of X) who will cherry-pick metrics that suits their perspective.
Government departments dedicated to planning are often required to publish their guidance/research publicly making it difficult to get away with cherry-picking metrics.
Having a team of professionals who understand the basics of our economic system tackle the problem seems to have a higher probability of success as compared with business leaders who often lack formal training in statistical methods.
There are more arguments but this should be enough to prove this false equivalency.
Second, decision making is always inherently political; it cannot be assumed away by throwing "professionals" at the problem. This is a consequence of "value" being entirely subjective. When economics was "political economy" (or even a field within "moral philosophy"), the field understood this. But that arrangement didn't give the professionals and academics enough status or prestige, so American economists in the late 19th century who had studied in Bismarck-era Germany rebranded themselves as "economists" who were practicing a form of science not too different from, say, physics.
Third, business leaders have to make a profit and maintain positive cash flows in order to sustain themselves. If they don't, the business dies. This is an extremely hard constraint that disciplines any business leader regardless of statistical literacy. States, on the other hand, can always subsist on the use of force to extract revenue. Central planners can publish metrics till they're blue in the face, but what constraints are they under to make the metrics any good?
Your belief is that governments never do this, that government decision makers do not make choices off the their own personal whims??
https://www.youtube.com/watch?v=idb_qsAAe1c
"I'm a nuclear physicist. Before you were deputy secretary, you worked in a shoe factory."
"Yes, I worked in a shoe factory. And now I'm in charge. To the workers of the world."
> What's as big as a house, burns 20 liters of fuel an hour, puts out a huge amount of smoke and noise, and cuts an apple into three pieces?
> A Soviet machine for cutting apples into four pieces.
How much steel was produced yearly, how much coal was produced yearly, and how the old bourgeois Czechoslovakia (1918-1939) was so much worse off in comparison.
What was NOT trumpeted out loud:
* we were seriously behind in production of many consumer goods and in their quality, * some parts of our industry could not compete with Western products on the account of being obsolete - especially electronics.
I'm not sure how far off this is, but socialist propaganda always reminds me of corporate BS I've seen over the course of my career. At least with a corporation, I can leave.
At the end, Wallmart finally regains its former functionality. The management proudly opens the windows that have been locked for decades. On the outside, they see drones flying everywhere, a personal teleport at every street corner, and rockets regularly flying to Mars. In despair, the entire project is abandoned.
It turns out that the government had a program that paid for these operations when performed on poor people. So the clinics would convince their customers that they needed major dental work, then would do the work and bill the government. The clinics were totally specialized just for this.
"Frontline" pinned the blame on the clinics, sort of "how could the do that". Of course, they're right. But blaming the clinics doesn't work, because the actual problem was the government set things up so that the rise of clinics to exploit the rules was legal and inevitable.
You're now faced with the problem of crafting a bureaucratic rule to define a "necessary" major dental operation. Of course, it's impossible, and the same problem as "management by metrics".
Not everyone chooses to do something morally wrong, just because "I'm allowed to! And it makes life easier!"
But plenty choose to do something morally wrong because "I have to put bread on my table, and I'm about to go out of business because I can't compete with everyone else doing the immoral thing". This is the mechanism by which the market eliminates ethics from our lives, step by step. If doing something slightly wrong gives you a competitive advantage, you'll take it or you'll get outcompeted by everyone else who do.
Again, it's about risk/incentive management, not about judging those who dare to go too far. The T-Rex will eat you even as you shake your fist at its unethical re-creator. Better to not merely hope that everyone chooses the friendliest route by virtue of human nature.
If we keep setting up situations where people have an unethical option, I’m not sure we should act surprised by them taking it. We can, it’s an option, but it seems naive.
The government had the option to think through their policymaking more fully. What accountability should they have for that?
If we award points for “trying, with good intentions”, and not for “actual outcome” we are in for a lot more bad outcomes from earnest triers.
Honest business by eastern Massachusetts standards would have you labeled a swindler in southern Indiana.
There are large swaths of the US where "if the option is there then there's no shame in taking it" and "if you don't cash in you're only hurting yourself because someone else will" are just how normal business is done.
For example, I have various dental problems. With each one, the dentist would lay out the options - effectiveness, how long they'll last, the price, how painful the procedure is, what the recovery time is, etc. There is no right answer. Different people make different choices.
(I've often picked an option that wasn't the dentist's recommended one.)
Medicine is almost never (Condition A => Treatment B).
The main point was that you can't simply blame the law for failing to 'replace judgement'. Most laws rely on some amount of common sense to be interpreted.
They get to take money from people backed by the threat of locking them in a box if they don't. They are responsible for making sure that money is spent extremely effectively.
Not only has it always been a business maxim, it's a maxim for pretty much every organization and 80% of the population. (I'm being generous, I suspect it is more like 95%.)
Trying to construct a society that relies on people being selfless most of the time is just never going to work.
It's an issue with all forms of health care. If a hospital makes money for each test done, the financial incentive will be to run a lot of tests.
I had a dentist pull that on me, I told him off, and went to another one. (He wanted to pull my wisdom teeth, I told him there was nothing wrong with them, and there still isn't 40 years later. Not even a cavity.)
That doesn't line up. In some countries, such as the UK, doctors are generally salaried employees.
With that said, I don't think the same is true of UK dentists.
This is not true. A GP practice is a private partnership like a legal chambers which bills the NHS for its services. This fact is strangely overlooked by people angry about “privatising the NHS”, it’s always been this way.
https://www.healthcareers.nhs.uk/explore-roles/doctors/pay-d...
Paid a salary by the private practice they work for, again just like a junior lawyer would be.
The obvious solution would be for the government to punish the clinics for acting against the spirit of the law. But that approach brings in a huge potential for abuse. We're in a bind.
If we can solve this, this will be the biggest breakthrough for civilization ever since the invention of hierarchical structures many millennia ago. Hierarchies and codified laws were what allowed us to stop living as isolated groups of ~150 people[0] and go on to build a global civilization. But these came at the cost of not being able to exploit our innate social skills to cooperate with each other. Bureaucracies, rules and metrics are thus an inefficient and barely adquate mechanical substitute for trust.
If we could somehow scale trust, scale social skills, so that people would naturally respond to incentives that are uncomputable and not expressible through bureaucratic apparatus - like "don't do stupid shit", or "really, don't cheat" - we'd enter a golden age.
Alas, I have no clue how to do that.
--
[0] - Yes, the Dunbar's number. Past which you no longer have the case that everyone knows everyone else, and interpersonal dynamics stop being strong enough to ensure mutual cooperation.
Free markets are remarkable in their ability to get people to cooperate in their mutual self-interest.
--
[0] - https://en.wikipedia.org/wiki/Barriers_to_entry#Examples
The problem here is that there is no way for society at large to send corrective feedback, "holy shit you're totally abusing the rules and what you do is immoral", and have it actually impact said clinics' behavior. This mechanism exists naturally in small groups, but money and scale destroys it.
Government paying for things is not the free market in action.
This is a recipe for market failure and not something I'd describe as a "free" market, even if it is a market.
The government is spending its own money it collected in taxes[0]. Whether it is more like a thug or more like an embodiment of the collective will of society is immaterial. It's just another customer on the market.
> This is then made worse by the fact that the party who receives the services purchased (the patient) is someone different again who has no skin in the game at all.
This is the exact same scenario as an advertiser paying a company to expose me to advertising; I have no skin in that game either, and I'm subjected to something I do not necessarily want. But nobody would say it's not "free market". Hell, in the case of the clinics, the situation would be identical if the party paying for the procedures was not the government, but a billionaire philanthropist.
In fact, nothing about a free market precludes such scenarios, where the choices available to a buyer are dependent on the deals made between the seller and some third parties.
--
[0] - The concept that "government has no money" is silly outside the quite obvious reminder that government is mostly funded by its subjects.
Not convinced? Just look at the way government borrows: no sane person would sell her future like that. Also, just ask a politician how worried he is about spending money "for his constituents". I know stadiums made for tiny towns, boulevards for undriven neighborhoods, and so on and so forth...
Look at what is going on in the stock market right now since the fed started buying assets in spring.
Now if you want to talk free markets in general, then I want to point out that in the real world, a free market cannot thrive without some sort of government that ensures social stability and enforces contracts. Without such government, the market would quickly degenerate into trading violence alongside currency, and either disintegrate or... end up producing a government. Ultimately, "private enterprise" and "public governance" don't have a clear boundary between them - they're two modes of the same phenomenon, people negotiating with each other.
I am not proposing to get rid of the government, that leads to anarchy. A government is required to uphold the rule of law and, for markets, to sustain contracts. I would also argue that they should tax externalities. The trouble starts when governments go beyond that and start regulating and influencing the markets, ending up corrupting them.
There's of course lots of debate about what the right policies to manage and mitigate these are but it doesn't mean that markets aren't a good mechanism just because there might be trade-offs or imperfect solutions.
I'm also not saying markets aren't a good idea. Just I don't believe they're the core of the solution to the problem I outlined upthread.
Why would it matter if the corporation is the government or an insurance company? Also, are you implying that you can't have a free market if you have insurance companies?
But it's far from perfect, especially in the US (at least as I understand the American system).
By this I mean that instead of the government paying each hospital (or in this case dentist) for each procedure, the government pays each hospital (or dentist) to keep a certain capacity for each kind of procedure available. The hospitals are obliged to provide up to that capacity at no extra cost to the government. Any demand beyond it, goes into the backlog (and turns into a political issue).
Surely I'm missing some downsides…
In the long term there's still some incentive to take on extra clients for the care providers so they have a better negotiating position for their next contact.
I guess that the government did not "set things up so that the rise of clinics to exploit the rules was legal and inevitable". I am going to guess that individuals, knowing that that was not the intention of the government aid, decided they could get around it to profit.
edit: typos
edit: retracted: (sometimes leading to more expensive measures needed down the road) — see replies
We already know the thing about diabetes, to the point that even private insurance companies will throw money at compliance because it’s cheaper than care. Outside that, vaccinations, and maybe a few more specific cases, the conventional wisdom of early intervention saves money is false.
https://en.m.wikipedia.org/wiki/Oregon_Medicaid_health_exper...
I think that's a heck of an important thing to mention here. Costs didn't change much, but health was improved.
We can blame the owners of the clinics because we have a right to expect people to to make morally correct decisions. We will never invent the perfect system of laws to incentivize everyone's good behavior. At the end of the day, society only functions when the majority of people are at least somewhat selflessly contributing to it.
We can blame those who devised and implement the program because we can expect them to do more auditing as well and ensuring the program isn't gamed. People in government are responsible for spending taxpayer money wisely.
I think the answer is incentives. The government had the incentive to “help the poor” - so they did. The clinics had the incentive to “fix teeth for people in need” - so they did.
Just like in software development, so much rests on properly defining the desired outcome.
What exactly did you just prove by this overly generalising statement?
The idea is simple: a government that does less costs less, so it needs to collect less taxes. That means people just over the edge could pay for their dental work instead of the government and all the overhead of having a third party involved.
It doesn't help those completely unable to care for themselves, but those who are close are no longer pushed over the edge.
"The clinics had the incentive to “fix teeth for people in need” - so they did", in the case at hand, being the procedures unnecessary, the incentive was not "fix teeth for people in need", but "increase profits".
You are asserting that the government’s policy was sufficiently specific regarding “medical necessity” that would clearly show these claims would not qualify. If that were the case, then the government itself should not have paid out those claims in the first place. The fact that they did pay the claims (at least until someone blew a whistle) suggests either that their policy was vague enough to not be so enforceable, or that their review and enforcement was lax (or both).
That’s my point - if the policy and enforcement had been clear from the get go this wouldn’t have been so big a deal. One or two bad apples try something and get sanctioned and the whole thing stops being an incentive to bad actors.
We agree that something changed in the situation such that this is no longer occurring, right? What was that change, and why was it not so in the first place?
If there also is a government program that reimburses for X in almost all cases, then you could see the emergence of clinics that "exploit" the rule even if everyone involved believes they are behaving ethically.
There are feedback loops that then exacerbate the problem- for example a successful clinic might bring on additional dentists, who despite misgivings will be influenced by existing practice at the clinic and begin self justifying the procedures to conform to the senior partner. After all, they clearly know what they are doing, right?
While I have no idea if that's what happened here, it is well with in the realm of possibility. The government does bear some responsibility to ensure that thier programs and rules provide sufficient oversight and feedback to keep fringey participants within the bounds of what the government considers acceptable.
A better example would be if a Sr Director of a rental car company shifted sourcing of thier repair contractor. If the new repair contractor consistently did way more repairs then the rental company needed, should the director of fleet maintenance be held responsible? Absolutely.
We don't say that an ATM mugging victim bears some responsibility just because a criminal knows that such people will almost always pay out.
possible actions:
politely ask the general public not to mug one another
increase penalties or rate of apprehension for perpetrators
encourage people to minimize their ATM visits, not walk around with debit cards, etc.
give people money so they don't feel the need to mug other people
The moral failing is on the clinics, sure, but if you want to fix the problem, you have to look at the government.
it's not rocket science. Poor people don't have the money required to go to the dentist and properly take care of your teeth. Since having a healthy populace is a good thing, we say "okay, we're all going to pitch in so everyone can be as healthy as possible."
If people start to take advantage of that for personal gain (which we know there are always people who are going to do that), we need to have a plan in place for dealing with those assholes.
The presence of assholes in society shouldn't mean we don't even try to take care of each other. It should mean that we have plans for recognizing when someone is an asshole and taking measures to change that behavior.
This is going to be the problem with every system of mandatory funding. Doesn't matter if it's housing, dentistry, food stamps, etc. People were abusing the disability program, so we made a bunch of hurdles to get funding. Now there's companies who train you how to get over those hurdles: what to write on the forms, what to say to your doctor, etc. And guess what? The companies that train you what to say to get disability then get to bill the government for this new service they provided you. I'm sure it's easy to imagine their incentives.
How would ending food stamps stop you from taking care of other people?
If I'm acting on my own, my reach is very limited. By using the government, we have the potential to make a much larger impact, and in this case, create the expectation that no one should go hungry in this country. We're not there yet, mainly because there are many people around here who don't believe that expectation is valid. I'll fight them for the rest of my life.
You can't write all laws like we're living in a Mad Max dystopia - some laws will need to assume that other laws are functioning correctly in order to be practically implementable as a law - just like software has dependencies that must uphold some invariants for the interface to work. The issue isn't subsidizing healthcare, it's unethical professionals abusing our trust.
Generally true, but the judgment call on "is this patient better off by being left alone" vs. "is this patient better off by having a surgery" is not easy, especially in our culture which worships heroic medicine. Patients often demand that something be done because they are primed so from the popular media they consume.
This can be even seen in the end-of-life care. Too many families demand that their dying grandma is given expensive and invasive care to live three weeks longer, even though its not just pointless, but outright cruel.
That's why it's a problem with professional ethics in healthcare first and foremost. Nobody in the bureaucratic chain is qualified to make or judge medical decisions and them trying would only be a monumental disaster because they don't even meet the patient and they're probably not medical professionals themselves. Hell, the patients and their family are rarely qualified to make the decisions - just look at what doctors chose for end of life care versus what their patients end up doing.
The system doesn't work if we don't police the professionals making the decisions at the point of care. The trick is not overdoing it like we did with opiates, scaring doctors from prescribing them even for patients genuinely in need.
A new "moral heroes" national holiday celebrating those who uncover fraud. More resources dedicated to investigating claims, more leeway to fine corporations for bad faith activity in order to increase the incentive to blow the whistle.
There was a recent UK scandal over the food boxes being handed out as a substitute for free school meals while schools are closed; it was immediately obvious to anybody who has ever shopped for their own groceries that the intermediary company was taking the £30, skimming off most of it, and providing a deeply inadequate result: https://www.dw.com/en/food-parcels-for-kids-spark-outrage-in...
(I note that even in the British NHS, dental and optical services are not completely free at the point of use: https://www.nhs.uk/nhs-services/dentists/dental-costs/unders... )
> Chartwells, the school catering company that supplied the package, claimed the photograph showed five days of free lunches, not 10, and the charge for food, packing and distribution was £10.50 not £30,
https://www.theguardian.com/education/2021/jan/12/food-parce...
Anyway, the campaign got the policy changed; Starmer went and looked at the DFE guidence to check that the expected parcel really was that bad. https://hansard.parliament.uk/commons/2021-01-13/debates/C80...
Yes, and it's evidence-free nonsense, because many of those people are either not earning at all, often for very good reasons, or earning below the lower tax threshold. You might be able to make a case for abolishing sales tax/VAT, but at least in the UK food is already exempt from VAT.
One definition of poor is that people do not have access to enough money to meet their needs.
And, poor people cannot meet the needs of getting more money: tasks such as education, job search, etc, are limited by an immediate need to get rent, food, and warmth.
If the government "gets out of the way", the safety net of being near-guaranteed to cover rent, food and warmth, will also quickly get out of the way, exacerbating poor peoples' problems.
Everybody will try to make their lives better. Anyone who isn't careful will end up spending so much on little things that they can't afford the basics at the end of the month. I've been poor (for the US - a much higher standard than poor elsewhere!), and I have no desire to go back.
I will agree the poor who cannot take care of themselves are a minority. However I maintain that given money to the rest won't help because they will spend it on things like a nicer apartment. Worse I fear that their attempt to get a nicer apartment will make rent go up and in the end they spend more money on the same apartment - thus making that extra money an indirect handout to the rich.
Rent-seekers will attempt to extract the maximum they can get away with. The solution is controls to prevent that. Anything that is essential (rent, utilities, food) should not be subject to the whims and greed of private interests.
Plenty of countries' and cities' policies are successfully curbing rents and landlordism - why are you ignoring empirical evidence to make your point?
And yet... the USA ranks higher than India in homeless people per capita, and also for street children.
> However I maintain that given money to the rest won't help because they will spend it on things like a nicer apartment.
And an appropriate level of heating, and healthier food, maybe. What's wrong with that?
Actually, blaming the clinics does does work, or at least is appropriate. They're the ones that are doing unnecessary work. Just because you can do something does not mean you should do it.
I person can cheat on their spouse, but should they? Was it the spouse's fault that they let the person go on a business trip/conference unaccompanied? The cheated-on spouse should have know better and set up a system to prevent it from happening.
Blaming the government here is a bit like blaming a life insurance company when someone murders a relative for the insurance money.
I've had some dentists do the "mouth mining" kind of thing.
I have my four wisdom teeth, and one dentists was immediately "We should schedule you to have those wisdom teeth out." He's not the only one that asked, but "why should they come out?" always has these vague answers, like "might cause crowding".
So, I've had my wisdom teeth all my adult and they are sort of a litmus test for dentists.
I've heard navy seals require them out, I thought for some deep water diving reason.
commonly known as Goodhart's law (https://en.wikipedia.org/wiki/Goodhart%27s_law) although credits should go to Marilyn Strathern for this formulation.
edit: so apparently this anecdote was in the article... I guess I just gave away that I commented before I read the article...
Reminds me of the cobra problem in India. After England colonized India, they noticed that India had a cobra problem, so they created a bounty for cobras: bring a dead cobra, get paid. So people, of course, started farming cobras to get paid by the English government. Once the English government figured out what was going on, they scrapped the bounty program and the cobra farmers released all of their now-worthless cobras into the wild... increasing the cobra problem.
KPI, KRI, KCI, etc. Truly a heap of metrics that eat out the time of middle management, and consequently of the people down the hierarchical pyramid. It over-allocates resources towards resource management itself rather than towards products themselves.
There is a drive to justify your existence, and that of your fellow workers, when your company becomes large enough that it has bureaucratized and, moreover, acquired some form of legacy.
David Graeber in Bullshit Jobs mentions that two categories of BS were the box-tickers and the taskmasters. They are evidence of this management by metrics.
It's all about budget: gotta spend that hard-earned budget to find ways to save budget on your budget-consuming core products... to make sure your budget allocation is the same next month or year.
You mean, it provides lots of avenues for the management class to provide cushy jobs for each other, at the expense of the real workers. Most organisations could ditch most of their middle management without skipping a beat.
metrics that are reported to individual contributors (e.g., toyota production system, agile done appropriately), on the other hand, are fantastic, for managers and workers, as long as it's well-understood that the metrics are not the product, they're simple signposts (indirect/incomplete measures of holistic qualitative desires like efficiency and quality).
by the way, (bad) bureaucracies are created by the drive to regiment and order messy human processes that leads to over-reliance on metrics, not people trying to justify their existence.
bullshit jobs is a bullshit book written to appeal to our fears and frustrations, not to elucidate us with actionable information.
However, most metrics I have encountered and helped impleme t in the past, especially in the finance industry, fall in that category. Boxticking and task-management because there is a mentality of cutting corners.
Bullshit Jobs is an essay, not a research paper. It points at a globally felt phenomenon in the workplace. People's impressions, no matter how subjective, still imply something is afoot.
Also, a book pointing at an issue is not obligated in providing a solution. Criticizing a system has its own merit. It outlines an issue maybe someone else will then be able to elucidate.
It is a baseless claim that public health is systematically ignored in favor of pushing the pandemic’s reproduction number down. Do we really expect public health officers to be untrained, stupid, and dismissed? Is that what we really are seeing? Do we really think that public officials are ignorant of the metabolic and social implications of having to drive stakes into society?
It is a fact that in functioning states, public health decisions are taken within an ethical framework of how to do least harm and preserve the most freedom, considering resource availability and longer-term effects of public health decisions, including the interacting feedback loops of the economy and public health.
While overreliance on small metrics is clearly bad, this style of off-hand ignorance of public service is not what we need.
It's not baseless when there have been multiple studies now showing lockdowns don't help, and even the WHO now recommends against them, yet authorities still push for them.
Not including deaths of patients who died outwith hospitals.
The best example are apples. When you breed apples to be 3% larger each year but taste slightly worse, you can weigh them and see that they got slightly larger. As you cannot accurately measure taste you will notice that they basically taste the same. After years of breeding you have giant apples with bad taste.
The author's proposition of using as many metrics as possible and talking to real people is a good solution to make better decisions.
I would add that for some decisions you should actually just focus on the few facts that truly matter. The classic example being a potential business partner with a bad reputation. If somebody is not trustworthy everything else probably doesn't matter.
Huge spread sheets with % and pivot tables and week on week breakdowns BUT no one could tell me much about what all that data meant. Apart from the fact that everyone had their own explanation it was just useless information to be fair. Churn, CAC, amongst various other 'hot' metrics.
In the end, most of these companies took a couple of years to realize that these metrics which Unicorns write amazing blog posts on are either BS or just not made for them. So much frustration was felt because a manager read a TechCrunch article about how X (with 100's of millions) did this and achieved something spectacular.
It almost never works out and then the leaders blame teams for incorrectly following whatever the process was rather than accepting that almost every org is somewhat unique so adjustments to strategy need to always be made.
One of my consulting roles got terminated early (1 week before I was about to terminate it myself too) because a Growth Hacking agency with a loud speaking founder turned up, told the founder that my work was not good (an agency saying that there's problems across the board - who would have thought that huh?) only for them to be fired 1 month later and an outcry across social media for 'real growth hackers' and not promising the world from that same founder.
Bearing in mind this founder also had the latest Apple products because it was a must...
This sentence also describes the vast majority of startups - "It’s when we imitate behaviors without understanding how they work in the hope of achieving the same results. Naturally enough, it’s not something that works."
Whilst probably not as beneficial as predicting behaviour, there could be a startup version to find the origins of buzzwords. Then try to predict how management will potentially push that company forward using these buzzwords or references to external sources.
Because they did this and (coincidentally) achieved something spectacular :)
1. Look at a complex and confusing reality
2. Fail to understand all the subtleties of how the complex reality works
3. Attribute that failure to the irrationality of what you are looking at, rather than your own limitations
4. Come up with an idealized blank-slate vision of what that reality ought to look like
5. Argue that the relative simplicity and platonic orderliness of the vision represents rationality
6. Use authoritarian power to impose that vision, by demolishing the old reality if necessary
7. Watch your rational Utopia fail horribly
[0] https://smus.com/books/seeing-like-a-state-by-james-scott/
That's an elegant way to articulate a criticism I've had of many technologies that I have been unable to state so clearly. All those file formats, for example, that seek to replace XML. Think of JSON, YAML, and TOML. They all claim to be simple, but end up too simple in practice and cause just as many issues as they solve. Comments, schemas, and namespaces were there for a reason, they weren't just a "messy design" awaiting a visionary to clean up...
> 6. Use authoritarian power to impose that vision, by demolishing the old reality if necessary
Unfortunately, in the cloud era, we're all faced with trillion dollar behemoth organisations that wield absolute power over their dominions. Us peasants have no hope of changing the vision they choose to impose. Suddenly, the well-established and well-known stable world has become "legacy".
The number may be the same, but the set of issues is different, leaving developers the choice if they'd rather work around the lack of namespaces and comments in JSON or around the opaque meta definitions and annoying syntax of XML.
And the popular vote was pretty clear on which design is preferred.
Oh, surely. The one largest problem of XML is that it's highly redundant. Nobody complains about those, just that it's verbose, had to write, and hard to locate oneself on the hierarchy when reading. All of those are caused by the redundancy. (Oh, and there is the issue that DTDs have an incredibly complex language that nobody knows entirely...)
As a rule, if it's a good feature for a programming language, it's also a good feature for a textual data encoding language. Ditto for bad features.
I base the value of a metric presented to management on the "and so what?" basis. In other words: how it helps them to better run the business.
There is the "annual objectives" kind of report: how much we did and how good we are. Meaningless.
There is the "current events" one (nr of incidents, etc.). Meaningless.
There is the "these are systemic, company-wise issues I need you to arbitrate over", or "we need to do that, but it means changes and incomfort". Actual decision making data, the worst to present because they show problems and not celebration.
I guess that this is the case of these organizations that do not provide green money to the company (aka. "cost centers"). i hate these metrics.
(please note that there are metrics that are useful to *operational teams* so that they realize there is a trend, or that some performance upgrades will be needed, but they should stay there)
And then after the quarter, “OK, Americans didn’t storm the Inauguration, but are you sure that’s because you so-called mitigated the risk with 20K troops? All this money, no attack, maybe we should spend that money on teachers instead of troops.”
In this space, lack of negative outcomes breeds contempt for the mitigation. (Hmmm...)
Interestingly, in the very long long run, that might even be right, as better education on civics is solving the problem at the left, earlier in the production line.
So now you have a new problem, which is to convince those deciders not close to the problem and disinterested in openly admitting gaps, that they need to both spend on the 20K troops now, and double down on teachers for a payoff 10 years from now, so that after that they don’t have to spend on the troops... best luck.
What I am trying to say that if I was management, I would be only interested to see things that are useful to the business.
If my business needs to be compliant with XXX, then a green status for XXX is what I need.
If there is a problem information security cannot handle because (whatever reason: no commitment form another organisation, no budget, ...) I want to know that.
Knowing that there are 7 attacks/sec or that there were 17 incidents would not help me.
This also means that I trust Information Security to provide me the right information and do their job correctly otherwise. This is usually checked by an anual external audits (at least for financial compliance).
What I do not want (this time in my real role in infosec) is to spend time providing meaningless data which management does not use anyway (but since "everyone" shows them, they want to see it too for a reason that escapes me)
To entice, here's a small excerpt from the abstract of [1]:
> The overabundance of information in late modernity makes the information society full of temptations. It tempts us into thinking that knowledge-as-information is objective and exists independently of human beings; that everything can be reduced into information; and that generating ever more amounts of information will increase the transparency of society and, thus, lead to the rational management of social problems. However, as argued in this paper, the information society is riddled with paradoxes that prevent it from satisfying the temptations it creates. More information may lead to less understanding; more information may undermine trust; and more information may make society less rationally governable.
[0] https://www.tandfonline.com/doi/abs/10.1080/713651562 (alternative: https://sci-hub.se/https://doi.org/10.1080/713651562)
[1] https://www.sciencedirect.com/science/article/abs/pii/S00163... (alternative: https://sci-hub.se/https://doi.org/10.1016/S0016-3287(97)000...)
- Metrics give everyone a false sense of objectivity. If you actually look at any stats in depth, you will end up making decisions about what to measure, and what the numbers mean. These words are actually the most important part of any study, but are always left out in favor of the graph. Want to know how many people died of covid? You need to consider things like population adjusted deaths. And adjustment for age. And a a whole host of other things to do with how the data was collected, and how it was presented. The economic side is even more messy. How much would GDP have been if we'd done this or that? There's no number that's the answer, but we do need numbers.
- You'll always care about more metrics than one. There's no generally objective way to decide how to weight those. You might have a utility function that decides this, thus reducing the problem to a simple optimization, but the guns/butter balance has still got to come from some kind of value that you express.
- The big question is rules vs discretion. You set up an elaborate measuring system (eg GDP) and you imagine that you have certain levers that you can move in response to whatever situation occurs. Part of this is that actors expect you to behave a certain way. But what happens if something you thought would work mechanically doesn't behave like you thought? Should you move your levers contrary to your previous commitment?
- Finally, there are the unmeasurable sensations. You have a team, it feels wrong somehow. You talk to people, get opinions, you get a feel for what is wrong. None of it is something you found in your LOC/user summary, or CI pipeline fails, or whatever.
Gut vs Metrics don't have to be either/or. Gut also has its own limitations - Daniel Kahnemann has documented this very well in Nobel Prize winning research. So what should one do? An organisation has to be self aware that there are no "perfect" metrics. Therefore, one should clearly state the limitations of a metric(s). Then, make a qualitative judgment based on the metrics and their limitations. You make the limitations, assumptions transparent and communicate the risks, uncertainties. Metrics are an input to decision making, they aren't decision makers themselves.
Unfortunately good decision making is a rare skill, and management is a common profession, so stuff like this will continue to happen.
Note that the prerequisite for making good decisions is the space in which to make a bunch of decisions, some of which will inevitably be bad, so without a cultural acceptance of potential failure and the right attitude towards it, things will never improve.
(Short interview with one of the hosts on the episode: https://www.fastcompany.com/90249924/reply-all-podcast-nypd-...)
For example, there are regulations that 100 ppm of pollution X is illegal, while 99 ppm of X is legal. Emit 99 ppm, and you're good. Emit 100 ppm and your plant gets shut down. Even worse, you can emit 99 ppm for 10 hours and you're legal, but emit 100 ppm for 1 minute and you're illegal.
A far better approach is to tax total X emitted, not the rate, and not binary legal/illegal:
100 ppm for 1 minute times $1/ppm = $100 tax
99 ppm for 10 hours times $1/ppm = $59,400 taxNot for air pollution. There the total emissions are of interest, not the rate. Car emissions, for example, are regulated for their rate of emission, not total emissions.
My proposal is to measure the emission rate, multiply it by the mileage driven in the last year, multiply by the tax rate, and pay the tax to renew the license.
It's not perfect, but it's simple and a far better approximation than the existing brain-dead method. Instead of incentivizing drivers to just pass the test, they are incentivized to minimize the total pollution their driving emits.
I agree that's bad. Far better to specify the goal, not the process.
A plant shutdown at 100ppm will likely result in running the plant at 90ppm to account for the risk of breaching the limit. But if I only pay an extra $100/min when run at 100ppm I'm better off running at 99ppm and just paying the extra tax when I occassionaly exceed 100ppm.
In your example, the tax incentivises you to pollute to the point you can afford. It might be cheaper to just pay the tax than actually make the required changes to your process.
However if the regulation sets a strict limit you don't have a choice, you have to make the changes necessary to comply with the regulations.
To the author's original point, it is important to look and what you are trying to achieve and why. In some cases tax incentives/penalties will make sense. In others strict limits/bans will be required.
The idea is to increase the tax rate so there is enough incentive for the total aggregate of pollution across all the producers to be reduced to an acceptable level.
Good project management means being better at finding the right metrics - or at least recognising that our metrics aren't good enough, often because we haven't all of the information - why are we hiring a new developer, why is this piece of software being developed etc.
A dumb but easy to reach metrics helps if you’re employee that wants money on next evaluation.
An ad targeted at the wrong people still brings money to the ad platform, helping find the good targeting mechanism is of low to negative ROI.
And so on. As long as it’s an adversarial situation you’ll never reach that appropriate target that needs to be shared among a number of people so they move in the same direction.
I think metrics are unavoidable for management, but they shouldn’t be on what you actually strive for, otherwise you’re just sending wishes to a monkey’s paw.
The thing is just that it is not the case. It in intricate networks of relations, aspirations, missions, ambitions, needs and so forth.
Metrics should underlie that fact, not overlie it. Ie. they can be used, just in moderation.
I've spent sometimes hours in talking to clients on how they should first focus on metrics that help them ask the right questions.
This is prevalent everywhere, Scrum-masters holding meetings to discuss the scrum velocity.
My take is that quantitative metrics are incredibly useful when paired with strong leadership, a clear mission, and vision statement that give human context to the metrics. That way whenever “the numbers” are brought up, the context and focus are around goals that really matter (are we creating a product that is meaningful and making people’s lives better?) versus cherry-picking and gaming the system to meet KPIs. Quantitative metrics are a dangerous tool since numbers are so easily digestible, so special care should always be given to understand exactly what the metrics are holding people accountable to.
An example that comes to mind in personal life is fitness. Weight is such an easy metric to get your hands on... doubly so since you can quickly judge weight by looking at yourself in the mirror. It’s really easy to tunnel on trying to lose weight! The activities (both physical and mental!) encouraged by fixation on weight can be incredibly destructive. If instead I first start with a strong vision statement that aligns with my values: My body is my most important possession; I want to keep it healthy so I can live a fulfilling life. In the context of that purpose weight is an important metric to keep in mind as a risk factor, but it is also clear that there are a multitude of other metrics that will allow me to measure, stretch, and grow towards a productive end.
The next group makes subtle mistakes. They’re the ones who have learned how to use metrics and now they have a hammer, everything looks like a nail. I see this a lot in less senior data scientists. They know how to drive with data, and so everything must be done that way. But they make the mistakes this article gets at.
The group I like takes a middle ground approach. We recognize that certain network effects are so coupled that you can power maybe one proper experiment on it at a time, and even those experiments might take too long. Other experiments take too long simply by virtue that the thing you want to measure might be long term behavior. So you experiment on what you can, get everything you can by proxy, and make sure to at least measure health via the metrics that matter, but you don’t try to base every decision in something measurable simply because it’s measurable.
> A resume will never tell you if a person is genuinely interested in uncovering unknown truths about nature. But a 30-minute conversation will.
There is a technique that's successful and widely used. It's somewhat unfashionable to talk about it in today's political climate, it requires doing some actual work, and it isn't a "formula" - but it works.
The technique is exactly what TFA describes: talk to the candidate. Apply a domain-specific bozo filter (like FizzBuzz) if you must to narrow down the pool of candidates, but ultimately what you need to do is to get multiple people from the team for whom you're hiring to talk with the candidate for some hours, in a semi-structured way.
Cue in accusations of bias. Sure, it's biased, but what's the alternative? Pretending you can codify worker competency and fit as a set of cheaply testable metrics that are independent of company/team specifics? Such process doesn't exist, and will never exist, because the problem is AI-complete. Instead, why not trust that the people on your team are decent human beings looking for their own interest?
And in fact, a lot of companies do just that and happily chug along. But admitting to that doesn't make you an industry thought leader, nor can it be packaged into a service and sold.
Your post hit really hard, because a) Sole focus of my team IS MRR. b) I wrote something very similar. Sorry if my comment is a little maendering, I am a dad as of 7 days and what they tell you about sleep is no joke.
I want to plug in one of the less famous Drucker quotes:
> Working on the right things is what makes knowledge work effective. This is not capable of being measured by any of the yardsticks for manual work.
It's true that "What gets measured gets managed" by the converse is also true - if you focus on one metric only, the rest will fall into disarray.
It's good to keep your eyes at the gauges when you are flying, but just as the map is not the territory, the measurement is not the real thing.
People love metrics, because they give them an illusion of control over a complicated, real world, messy reality. The core of the problem is unfortunately human bias to simplify. We as a wider tech community are contributing a bit introducing yet-another-dashboard.
But as you rightly point, the metrics people pick are the ones that are easiest to measure. NOT the ones worth measuring. In fact I published a post quite similar to yours: https://piszek.com/2019/11/24/metrics/
To broaden the theme, I would even argue that Taleb is complaining about roughly a similar set of problems: Gaussian curve is an oversimplification as well and has caused a lot of suffering when misapplied.
Despite all of these caveats, there is a benefit to having a focus (like MRR you are mentioning). Having a singular focus can be transformative and can make one developer more effective than 30-people team with no real direction.
In that sense, there is benefit to metrics. As with all real-world things, I think the only solution is a dynamic equilibrium, where you change from one approach to the other to cover your bases.
As I wrote in the beginning of this comment, the Team I Lead (we build simple ecommerce tools) was tasked with maximizing MRR and the side-effect of having that focus was that we ignored the one-off sales features that our customers wanted desperately.
This is completely false. Politics is the least metrics driven, mostly intention driven.
https://www.washingtontimes.com/news/2014/apr/25/sowell-good...
And what do they have in common?
Their core product got notably worse over time.
It’s interesting he doesn’t mention Apple. I have no insight but from the outside of say they have very different things they optimize for than revenue. Money merely pours out of their customer obsession instead.
The issue isn't that anyone believes that metric-driven management is a good option, it is that at the scale of 2,000+ people there are not a lot of alternatives. I could easily design a system that works at large scale ... but only by the employees and owners being the same people. Apart from that, the erratic us-them nature of stockholders and employees makes all the good structures untenable.
Lay out an org chart for 2k people then ask how long the manager has to work out what their reports are doing. Expand the hierarchy far enough that everyone has time to do a good job and then ask what the odds are that all the people in positions of importance agree on how things should be done by whom. To settle the number of disputes that arise in a big tree-like org structure there isn't have much of a plan B apart from metrics.
It would be nice to see more governmental support for limited liability cooperative business models. The organisational opportunities that would open up could allow for so much. Not to mention maybe workers could earning more.
I'd like to see the major changes in taxation and legal reform to favour encourage bigger risk/reward sharing with workers and reductions in paperwork and regulation to make it easier to set up companies with a diverse range of small, actively involved stakeholders.
I think it is notable how well American tech companies did with large equity handouts to their early employees. Maybe the model starts there. I don't have details, I do have a problem - to get great results, large complex organisations need a different model than salaried workers.
This isn’t true through. On this very site’s Startup School you will find lots of experienced entrepreneurs advocating for a metric-driven approach to growing your startup. Maybe not “one metric that matters” all the time, but Lean Startup is about measuring your company’s metrics and running experiments week-by-week to see what moves the needle.
Sure, slavish and myopic adherence to a bad metric will lead you astray. But management without metrics will lead you astray too.
A talk vs metrics is just higher dimension vs lower one. The information within a talk is complex. Different people received vary information from the same talk. So it's hard to managed by scale.
So a metrics system is just a trade-off for its scale.
A guy who lambasts metrics, but whose projects are all based on scraping metrics.
https://productexplorer.io/ https://newsletterspy.io/ https://gumspy.com/
That doesn’t mean you don’t have long term goals or nuanced objectives even if your measures are coarse, nor does it mean your measures stay the same.
I think optimising for measures focuses activity , which again doesn’t mean you have to optimise for the same thing all the time using the same measures.
1. Taylor metrics - The ones invented by management. 2. von Braun metrics - The ones invented by the guy that put men on the Moon. These are everywhere it really matters.
Monthly Recurring Revenue, if anyone is wondering.
What a joke, all of these posts have the same conclusions: don't deal in absolutes.
> Google’s search results are dominated by ads
Googles core product are ads, not search engine!
There was a time before Google. There were other search engines with large market shares. Google completely destroyed them through superior search results.
An alternative search engine that actually works better as a search engine would be an existential threat to Google, and they know that. Because there is very little that prevents people from switching search engines - no network effects, nothing really except default settings and laziness.
Adsense controls like 80% of ad market. Biggest thread to Google are walled gardens like Facebook or Amazon. Search engines like Bing and Duck are not even on their map.
> dominance ... which is based on the quality of the search results
Any source for that? Google search already provides inferior results in many areas, but it is still dominant. Google also pays billions to Apple and Mozilla to be default search engine.
And if threat emerges and gets like 30% of search market share, Google just dials down ads and make their search experience a bit less crappy.
Only about 14% of Google's advertising income is from AdSense, the rest is from ads on Google's own properties, and the vast majority of that is from search.
> Biggest thread to Google are walled gardens like Facebook or Amazon. Search engines like Bing and Duck are not even on their map.
I guarantee you they are. Facebook and Amazon are threats because they are established with massive userbases so anything they do can quickly have a big impact.
But I absolutely guarantee that competition from a better search engine is very much on Google's map as an existential threat, just not an immediate one. The know they cannot stagnate in that field and are continually investing into into research to stay at the bleeding edge.
> Any source for that? Google search already provides inferior results in many areas, but it is still dominant.
How do you even quantify search result quality? My personal experience from trying out other search engines is that Google is definitely superior, and better search results was most definitely what got them to dominance in the first place - Altavista and Yahoo and Excite weren't even in the same ballpark.
> Google also pays billions to Apple and Mozilla to be default search engine.
And that should tell you all you need to know about how essential it is for Google to stay the dominant search engine.
> And if threat emerges and gets like 30% of search market share, Google just dials down ads and make their search experience a bit less crappy.
At that point they would already be fighting desparately for their life, because "just dialing down ads" translates into an immediate, proprotional decrease of revenue from their (at that point already compromised) main cash cow. 30% less ads on 30% less searches would immediately mean Alphabet as a whole is deep in the red.
And if a competitor already managed to take substantial market share from them, making the experience "a bit less crappy" is hardly going to be enough to turn things around.
No, Google really must be (and is) much more proactive in this: keep investing in research, keep on the lookout for any company that looks like it's doing something that might someday in the future impact search engine technology, and just buy it.
My point is that search is not priority for google anymore. 15 years ago it won because it had clean simple UI. Now its bog down with ads. Duck and others offer much better experience.
I am also not convinced by the inclusion of the R number in the list of examples - it is unlike the others in an important way. The problem with metrics usually arises when the metric is a proxy for the goal. For example, test scores are not the same as knowledge, and so using test scores as a metric can result in teaching to the test, such that the metric is no longer a good proxy for knowledge. But the R number is not a proxy - it is the actual thing we want to change. (Of course, there are many knock-on issues, such as how we measure that number, which do introduce proxies. Discussion of that would be a much more interesting article.)
Which actually points towards the real solution, which is not to simply rely on gut instinct, but to be aware of what you are measuring, and how it relates to what you want to know.
In it he talks about “Stalinist Capitalism” I.e. a capitalist society that is obsessed with metrics which optimize for arbitrary goals at the expense goals which are intangible or hard to measure. It focuses on the metric obsessed cultures that have developed within many private organizations.
Two examples come to mind, universities prioritizing publishing count vs publishing impact and lines of code written vs impact of code.