For example, a simple case: Say you go on vacation for a few weeks with a certain amount of cash to spend. Upon arriving at your destination, you immediately purchase some indulgence, and, hey, you're feeling better! Why not immediately keep spending as much as possible to maximize?
For example, a complex case: Say you're leading a country. You set up policies that, after a few years, seem to have led to a higher GDP than was previously expected. Does that suggest that the policies are leading to a better future?
In the simple cases, like the vacation-example above, it's easy enough to understand the scenario and what's going on. And we can imagine that, hey, immediately spending all of the cash might lead to poor consequences for the rest of the vacation, even if they display some quick-satisfaction at first.
But in more complex cases, like with a country's policies leading to a higher GDP, stuff can get trickier. We might say that it's the lack of a top-level model: unlike in the vacation scenario, where we were easily able to predict that there'd be a lack of money later in the vacation, it might be harder to say what else might be going on besides the GDP going up. And all other things held equal, presumably a higher GDP would be better than a lower GDP, and therefore all evidence points toward the policies being a good idea, right?
This happened with Brazil in the 1970's, when the oil crisis made Brazil believe that they were going to face a downturn. To overcome this, they borrowed money and went on an infrastructure spending spree. This made Brazil look like an economic miracle, growing when everyone else was struggling. This encouraged more banks to lend to Brazil. The problem is the money was spent on short term growth instead of things that would more systematically grow the economy over the long term. The government (and lending banks) was substituting year-to-year GDP numbers for economic health. Once credit tightened in the early 1980's, Brazil's growth plummeted.
This is of course an over simplification, but I think we are seeing similar problems in modern economies. People often cite China's GDP growth, but they don't balance that out with the debt they are taking on in order to finance that growth.
The vast majority of PMs I've worked with don't seem to understand this at all. If it's not in the metrics, it doesn't exist!
See also: https://www.theatlantic.com/technology/archive/2017/10/the-c...
But I admit the article gets to its point in a muddy and circuitous way, typical of business school anecdotes.
It's surprising to many.
Many people will decide they are "data driven" and act like this gives them infallibility and total correctness. Lots of people need to be told this.
- the data you are collecting may not include factors that are consequential to outcomes. Treating these unmeasured factors as inconsequential is hazardous. The piece specifically mentions this effect. This effect is not a surprise but yet many organizations fall into this trap so it seems to be worth mentioning.
- the difficulty of measuring some factors will result in their exclusion from the dataset. Some things are intrinsically hard to measure, and many organizations will as a result refuse to measure them, and make decisions without them. There needs to be a conscious and active process organizationally to resist this and find effective ways of measuring them.
- the difficulty of measuring some factors will result in easier but less useful proxies being used in their place. Same as above just with a somewhat different outcome. Organizations are often blind to this happening as they come to believe the proxy is as good as the real measure (or sometimes even that the proxy is the measure). A good industry example of this is clickthrough rates being treated synonymously with audience interest or content quality.
And a point I wish the piece made but did not:
- measuring something does not grant automatic understanding of the phenomenon and gives you no predictive power. It's one thing to quantifiably know the blue button gets more clicks than the green button, but that grants you no insight into why. Many tech companies fall into this trap - where despite investing heavily in experimentation their modeling of the product space doesn't improve over time, since they fail to take the step to convert observation to generalizable hypotheses that improve their model for the product and market.
This last effect IMO is huge in our industry, and is why the same low-level experiments are being re-done over and over again. This creates more product churn and reduces your product velocity - since the lack of proven product models means you're mostly flying blind and using ex-post-facto experimentation on live users to figure out what to do.
Experimenting on button colors is all well and good, but the end result of that data should be a color theory that explains what colors to use when, not forever A/B testing every single button color for the rest of time.