A good measurement culture where numbers don’t replace common sense
blog.promaton.com
blog.promaton.com
Comms. It always boils down to comms. Specifically, comms that drive understanding.
KPIs = knowledge.
But what you did, increased understanding.
And what I like to say is: "(Knowledge isn't power.) Understanding Is Power."
Knowing the dots doesn't mean you can connect them. And thus, "Understanding Is Power."
Anyway I think we are probably saying the same thing and discussing semantics.(and to be fair it is probably just me:) )
1. Determine whether there is supporting data for the proposed change. 2. If no data is available, implement tracking to establish a reference point. 3. Measure the reference point and ascertain what could lead to success. 4. Implement the change. 5. Review the new data and compare it against past values.
For each delivered feature, there could be an overview, e.g.
"Users can now print reports easier. On our user request tracking platform, this issue had 321 upvotes, and 5 Premium clients requested this as well. We placed the print button THERE because amongst the three designs, this performed 23% better according to METRIC. For more info on the experiments, see link or ask on #mobile-team. The feature was released for two weeks behind a feature flag, it performed great, and we made it available for all since last Monday. 5%, of users who visited the report page used this feature and we generated already 5000 PDF reports. Big client A already complimented the feature, and it unblocked the sales process with Client B."
At the end of each email, there could be an analytics overview of the product with overall useful metrics, significant changes etc.
This format also helps with people being on holidays, sick, etc.
I'm confused by your comment. How did you decide this was something worthy of fixing if it "doesn't show any good metrics"? If you can't quantify the issue in any manner, how do you determine it's worth doing?
Because sometimes some things without metrics are incidental to the actual thing you set out to do.
E.g. a large refactor that switches libraries which is necessary for your new service that give 10% lower latency. But that library refactor will need to be done and it will take 2 months.
This is especially important when metrics would not be expected to be available -- for example, if you're designing a nuclear reactor, you need to think hard about ways to prevent a meltdown in advance, rather than collecting meltdown statistics and then fixing the piping problems that correlated with the most nuclear meltdowns.
This is also necessary when the true metric that matters is very hard to evaluate counterfactually. For example, perhaps your real task is "maximize profit for the company", but you can't actually evaluate how your actions have influenced that metric, even though you can see the number going up and down.
And necessary as well when a goal is too abstract to directly capture by metrics, resulting in bad surrogate metrics: for example, "improve user experience" is hard to measure directly, so "increase time spent interacting with website" might be measured as a substitute, with predictable outcomes that bad UI design can force users to waste more time on a page trying to find what they came for.
All of these problems are faced by metric designers, who need to pick directly-measurable metric B (UX design metric) in order to maximize metric A (long-term profits) that the shareholders actually care about, but they cannot evaluate the quality of their own metrics by a metric, for the same reason that they were not using metric A directly to begin with.
(See also the McNamara fallacy, which parent comment is a splendid example of: https://en.m.wikipedia.org/wiki/McNamara_fallacy )
Impact on end users? Nothing measurable. Impact on developers? Frustrating and slows them down, but by how much? It's impossible to say. How often does it happen? Well, difficult to count what isn't there. Would fixing the issue lead to a measurable increase in stories completed per week, or lines of code written, or employee retention? Probably not, as those are very noisy measures.
Nonetheless, that is not a fault I would tolerate or ignore.
For instance, when executives contested the efficacy of our registration form for X reasons, it became incumbent upon us to ascertain its current standing. This involved computing the ratio of registrations to site visits and conducting an in-depth analysis of the errors that users encountered while interacting with the form. We achieved this by integrating Google Analytics events with error messages, among other techniques. We provided first a picture, then we proceeded to make the changes.
This systematic approach enabled us to gain a comprehensive understanding of the situation, facilitating the implementation of purposeful changes. Subsequently, we gauged the impact of these alterations using key performance indicators (KPIs) such as the registration-to-visits ratio, and the previous events tracking changes too to understand if we really made a difference.
Generally that was good enough.
Exactly. What does "performance improvement" even mean if you aren't measuring performance before and after?
Because naive senior vice president MBAs believe there never should have been bugs in the first place, and every fix is an admission that the bug existed at all.
Because every email is now a weekly contest between teams, and when hard times come the ones who reported on performance improvements and bug fixes will lose their jobs long before the ones who made the header the president's favorite color.
Because every email is your team trying to convince the company of your own worth and it's much harder to show a pretty screenshot of a performance improvement than a new feature (Graphs of before/after performance every week? Every week?)
Because bugs don't always follow your bullshit schedule made by a bullshit manager who is looking for a pat on the back by replacing bullshit A with bullshit B, instead of anyone anywhere in the company ever just trusting anyone.
- Improved p50 latency of endpoint /very/important/endpoint by 22% [attached mini-graph showing the change]
- Fixed this bug that had been reported by 37 clients in the last 30 days as seldomly affecting them
etc
Edit: I see the team is made up of 50 people across multiple teams. That makes a bit more sense.
It works great when everyone is delivering day-long or week-long incremental features that lend themselves to nice screenshots.
But then you slowly start accumulating a backlog of difficult tasks that everyone avoids because they won’t sound good for the week. Technical debt accumulates as people cut corners in the interest of getting their weekly presentations out.
You can theoretically avoid it with enough trust, but the trigger for our collapse was a round of layoffs where the people with the most visible features were spared while people working on important backend work and bug fixes got cut. After that, it was game over.
Also, if your team is optimising for the status report, your manager has already failed you.
That's why these kind of incentive structures are dangerous and why things like OKRs put such heavy emphasis on regular, company wide, failure. (I.e. they punish 100success rates)
With time and experience, you'll learn that those who can quickly churn out features and bug fixes are deemed extremely valuable to the company.
Let's say we have a wicked bug that has very little chance of happening. But if it does, it'll bankrupt the company, no questions asked. Spending 3 weeks fixing it is nowhere as impressive to the company as Joe who's churned out 10 features per week while your status updates are "Hunting for the wicked bug", even if you describe it it more details.
Yes, because this one task appears as a single line item for the manager's manager. They don't care for the complexity or consequences until the fire actually happens. And if the fire happens, they just blame engineers.
If the industry incentives were changed such that manager heads would roll for mass outages, managers would start appreciating big fixes.
That’s why devs who understand that survive layoffs and the ones who spend three weeks “hunting for an extinction level bug” don’t.
If you're providing a status report, you're optimizing for the status report. Period.
Engineers are not providing status reports. The manager is collating them. How is it any different to a sprint summary?
It's like imaginary tech debt.
The economics and tactics around technology has been revolutionized a dozen times over in the last 4 decades. Now, maybe a few rare individuals have kept up, but most likely, they all rely on outdated strategies from out of touch MBA programs & buzz words.
Its the same reason the market kills public technology companies' innovation and they rely on acquisition.
Its like gunpowder has just been invented and leadership still wants large formations marching against each other. "what if we invested in medical training and add washing our hands so we can keep more people alive?" "nah, more guns and marching"
I think its work experience that matters most with management. You need folks who are people oriented but understand the job that the folks they manage are doing, and the challenges that come with it, both obvious and non obvious, and can communicate the importance of such work to the broader organization
Ah, the cult of youth. It is cute, when it isn't killing startups with rookie moves.
It almost sounds like you think management is something like practicing law - like there's a set of rules that are revised on a schedule and they need CLE credits to keep them current.
> the Russian nuclear sub B-59, which had been running submerged for days, was cornered by 11 US destroyers and the aircraft carrier USS Randolph. The US ships began dropping depth charges around the sub.
Officers aboard had every reason to believe that their American counterparts were trying to sink them.
Cut off from outside contact, buffeted by depth charges, the most obvious conclusion for the officers of B-59 was that global war had already begun.
The submarine was armed with 10-kiloton nuclear torpedos and its officers had permission from their superiors to launch it without confirmation from Moscow.
Two out of three serior officers onboard agreed to launch, but unanimous vote was required.
I was specifically thinking of the idea that large formations being idiocy after the invention of gunpowder. Brett Devereaux's 4-part series "The Universal Warrior" had an excellent discussion of why a unit of soldiers might fight in large formation. https://acoup.blog/2021/02/12/collections-the-universal-warr... He's verbose, but thoroughly enjoyable.
The counter-theory is that as companies mature, they have to devote a disproportionate amount of time to maintenance. From that perspective, it's a risk-based approach to maintain the products and services that already have a market and instead effectively out-source the high risk operations of innovation. New companies don't have comparatively much to maintain, so they can focus on innovation. It's not quite such an either-or, but a balancing act. It's just easier to balance when you can outsource risk.
>Average CEO is 58 years old. Their first jobs were in the late 1980s.
I believe there's some neuroscience theory that may illuminate this. When we're young, our intelligence is more plastic and can more readily innovate. But as we get older it becomes more crystalline. What we may lose some of that novelty generation, we can gain in understanding the greater levels of experience. Experience is necessary for putting ideas in their appropriate context. I would argue that understanding context is more important at high, strategic-level positions.
Incidentally, marching large formations against each other has been the dominant strategy for most of Gunpowder's history. Only in the last 150 years did guns become lethal enough to make this a bad idea.
I can imagine a modern tech leader living in those times and giving the orders to charge the German panzer formation, sabers overhead.
I almost always tell management that by default I'm spending, say, 30% of my time on technical quality and ask if they'd like to dial it up or down temporarily because of a holiday or a deadline they can. I will track this and show it to whomever asks (e.g. their boss and boss's boss might be interested if they've explicitly asked for 8 straight weeks of 0% work on quality).
I think it ultimately comes down to human values. What's important to the founders and the team? Do they have a clear articulation of those values? KPIs are useful if they're grounded in values that the organization absolutely won't compromise on; KPIs should be a means to an end, not the end themselves.
Whether those values are "we want to make the most money" or "we want to make customers happy" or "we want a sustainable lifestyle for our team," KPIs will only help you pursue them, not define or prioritize them.
But modern organizations are quick to destroy trust on any whim.
I think I'm luckily grandfathered in due to working fully remotely for years before the pandemic, but if that happened to me I would absolutely start looking for a new job.
I've asked before if I can work remotely for a few weeks at a time and have been told yes, as long as I don't leave the state. In some places, just moving to a different county or city can trigger different tax requirements.
> State corporate or other business activity taxes can apply, if even a single employee is working in a state. In effect, if an employer did not previously have a recognized office in a state, but one employee starts working from there, this can trigger entirely new registration requirements and tax liabilities. It may be necessary to register with the secretary of state and relevant tax authorities, provide a registered agent address, and pay corporate and business activity taxes, sales taxes and employment taxes, including employee withholding. There are often state and local licenses and business permits as well.
https://www.adp.com/spark/articles/2022/06/implications-of-w...
So sure, if you’re a company doing only local sales. Or a company doing less than 3-400k of revenue a year, this is probably complicated. Once you hit ~1m in revenue, there is probably AT LEAST one other state you are paying taxes to. Adding states should already be a defined process (or being defined) and an employee moving there is only a one or two lines different.
And all states have long required employers to register with the state government for tax purposes if they employ someone working in the state, as well as comply with the state’s labor laws.
They back peddled pretty quickly and switched it to anyone within 10 miles of the office has to come back. There is only 1 person that close. He's the "office mom" and he's been in basically every day since the pandemic started.
Even if half the teams hadn't moved further away, there is no way most of us would have gone back to commuting 2-4 hours per day.
We used goals and velocity metrics on the highest performing team I worked on. This was also a high trust team that happily raised concerns and adjusted priorities/velocity/etc.
The goals and velocity were still extremely useful for getting everybody on the same page for what we were looking to accomplish and how long it would take. We needed to land in the general area, but never got caught up in meaningless drivel over metrics.
The problem is management wants consistency and expects an explanation when things change. I've found it a team is perceived as failing if they're actually realistic with goals and capabilities.
It is up to the team to decide on the best way to approach this, and what works best for that team, and the team is free to do the work as they see it, with the only requirement that if something does negatively affect the Milestone(s), it gets raised quickly and early, in case of re-adjustment.
This however, means:
- Product management has to be competent enough to present a relatively elastic vision that is not so concrete its essentially a waterfall, but not so vague that its unclear whats being built. Wireframes are usually a good signal here.
- Engineering Management has to be competent enough to communicate (or allow others to communicate) technical challenges that may be involved, and more importantly, what may be unknown, to Product management
- Everyone has to agree that demoing work is more important than talking about work, whenever possible
- Trust in everyone doing the right thing needs to be high, constant interference and meetings will kill this from working right.
It seems like such a natural division of labor that it appears everywhere. But I also feel like I've never seen a company explicitly optimize processes around it.
So I'm curious to hear any battle stories.
In a codified "trailblazers and firefighters" model, the trailblazers are much more obviously tied to the company's bottom line. Feature X is required to close deal with company Y, this software team built this feature. Meanwhile, the people frantically cleaning up the mess the first team left is only nebulously connected to the company raking in more cash, relegating them to a second-tier position despite having harder work.
While working at a big corporation we had a velocity initiative supposedly aimed to lead the company toward continuous integration.
"How long a PR stays open" was one of the KPIs in a dashboard.
I said: "Be careful with that!"
People started to close PRs and reopen new PRs with the same code.
Middle managers and sometimes the person in the division that was the point of contact for the velocity initiative were asking to do that.
The script measuring this KPI was improved to look at the branch name and the code diff. Result? People changing branch name and a change of EOL encoding in the new PR.
Learnings? B and C players with questionable ethics screw companies quite rapidly.
In this climate KPIs and aligning them with company values is futile.
Within a couple of weeks, scripts were circulating to auto-generate and auto-commit tests. If the JVM we were using had a bug in adding random numbers together, we'd have known about it very quickly.
That's about the time I decided I should move on. I'm glad I did. The company I joined treated developers like adults and developers acted like adults. And we had great automated test coverage.
Obligatory mention: https://en.wikipedia.org/wiki/Goodhart%27s_law
The company tanked six months after that, now it doesn't exist anymore.
There's only so much you can do when the management is hellbent on doing stupid things.
In that context, I’m not really sure what point you’re making, unless it’s just to share a personal anecdote. Are you implying that management shouldn’t have any quantitative measures and should only be qualitative?
If you sell, say, water bottles, you probably want to know how many of them you can sell at any given moment, in order to not overbook and have to reimburse people. In this case, keeping track of how many water bottles you do have in stock probably helps, keeping track of how many labels with funny jokes you can stick on a shipping box in an hour doesn't. But if you start tracking the latter and handing down bonuses and layoffs based on it, people will max that metrics out - at the expense of your actual stock capacity.
Quantitative measures are dangerous, especially in the hands of people who believe they are better than qualitative ones because they're "objective" or whatever. Because not only they aren't, but they are also better than qualitative ones at hiding their biases and soothing your own.
> Are you implying that management shouldn’t have any quantitative measures and should only be qualitative?
Many managers would do a lot better this way. They'd still make stuff up, but would at least be forced to admit it.
It helps to have someone with a hacker mindset think about the metric being designed so the obvious ways in which it could be games are taken care of and their own metrics/incentives are aligned with the company goal.
You can make an engine as effective as the laws of physics let you, but you can't solve the limits of thermodynamics. You can only do your best within them. Same with this, for maybe the same reason.
I scrap all incentivized metrics when working on something urgent (important and soon), which is often the case. If the metrics somehow incentivized, we'll start gaming it.
Now if a dev has part on production support and automated tests are purposed to reduce the workload for support then it has time slots for that, I bet everyone will start doing it.
How do you know what was really going on?
"He created a requirement that every developer commit two new tests to the code base every workday" seem a stupid requirement if you do not control the quality of the tests.
The same big corporation I wrote above had a goal of 80% code coverage reported on dashboards.
I saw people writing tests just to run lines of code, without effectively testing anything.
Others people were "smarter" and completely excluded folders and modules in the codebase with low coverage from coverage testing.
Code coverage percentage numbers on a dashboard are a risky business. They can give you a false sense of confidence. Because you can have 100% code coverage and be plagued by multitude of bugs if you do not test what the code is supposed to do.
Code coverage helps to see where you have untested code and if it is very low (ex: less 50%) tells you that you need more tests. An high code coverage percentage is desirable but should not be a target.
The real problem is again the culture.
A culture where it is ok to have critical parts of the code not being tested. A large part of the solution here is helping people to understand the consequences of low code coverage. For example collecting experiences and during retrospectives point out where tests saved the day or how a test may have saved the day so people can see how test may save them a lot of frustration.
But again, when you give people a target and it is the only thing they care about, people find a way to hit it.
> When I worked at a FAANG ~15 years ago, a new VP came in and heard, correctly, that our group didn't have enough automated tests.
If a team has identified a weakness in testing and transparently reported it, presumably with the intention of making it better, then why would we assume that setting arbitrary targets based on some metric with no direct connection to the real problem would help them do that?
if team does not have automated test, but still manages to deliver working software - maybe tests are not adding as much value as VP thinks?
the most important is feature delivery, and integration test, not automated unit test where you test getters and setters with mock dependencies - absolutely useless busywork
Also "not testing a lot" is not a chesterton's fence. "not testing a lot" can't be load-bearing.
Schedule isn't always the most important thing either. It's possible delivery the software may just mean you've been rolling the dice and getting lucky. The Boeing 737MAX scenario gives a concrete example of where delivery was paramount. It's a cognitive bias to assume that "since nothing bad has happened yet, it must mean it's good practice"
The culture lead to fake tests instead of adding tests that were legitimately lacking.
Is that so different from saying they weren't acting like adults?
The phrasing could be called dismissive but I give that a pass because it was mimicking the phrasing from the post it replied to. The underlying sentiment doesn't seem wrong to me.
Or, conversly, I've been in charge by really awkwardly testable code.. which ends up being really reliable. Plugin loading, config loading (this was before you spring boot'ed everything in java). We had almost no tests in that context, because testing the edge cases there would've been a real mess.
But at the same time, if we messed up, no dev environment and no test environment would work anymore at all, and we would know. Very quickly. From a lot of sides, with a lot of anger in there. So we were fine.
However, I don’t think mindlessly creating tests is acting in good faith. It’s bordering on malicious compliance. I doubt they were thinking they can just knock out that metric so they can otherwise create better test coverage. (The OP conceded their coverage wasn’t good). Better employees would work to create a better understanding/goal. All of that points to some cultural problems.
If I was a developer there, I'd totally started adding those generated tests. I have a job to do (ship products) and you put in some stupid requirements which actually interfere with it (since my time is limited I can either work on tests or my daily tasks like developing new features or fixing bugs), but we both know that if my primary job suffers, I'll pay for it. So the best solution, from my point of view, is the one that takes that requirement away and lets me go on actually working.
When we had a similar problem in a previous company, we just created an epic, assigned a couple of people to it and have them churning out tickets for specific improvements, like "Class X has a coverage of Y on this method, add tests for the missing execution branches", which were clear, non-generic and fully integrated in our flow. If anybody complained about our velocity or whatever, we could show them the full trail which lead us to choose to work on that ticket and how much we spent on it. The coverage issue got solved in less than a month.
If I were a manager there, I probably wouldn’t hire you. You want people who are actively solving problems that matter, not just automatons beating their chest about how stupid everyone else is while they add to existing problems.
But if the devs at the org all (or even substantially) tend towards malicious compliance, that's a sign that something has gone wrong, relations-wise
Any smart manager, having to deal with this kind of policy, would either push back or approve any way to game it and then get the work done. But I doubt that a company which allows this kind of BS can retain smart managers for more than a couple of months.
It’s telling that you show such dichotomous and defensive thinking.
I just hope to never work with you and, especially, never use anything you helped building.
I made the point that the metric used (raw number of tests) is a bad way to enforce a good goal (higher code quality). I also made the point that metrics aren’t inherently bad, but they have to be chosen judiciously.
You seemed to take those two points and extrapolate an entirely different story as a personal affront and apply motives that, frankly, reads as a bit unhinged. So i also hope we don’t cross paths, because in safety critical code development where I come from where bad practices and toxic teammates can kill people.
Also, with an outstanding company culture, KPIs aren't really necessary.
So, when would they be useful?
I'm not as negative on KPIs as the previous line suggests though. They can be useful to shape direction when used carefully. But don't make them too long-lived, discard and create new ones as soon as they become gameable.
If end of sprint came and you weren't done, the manager would close out the ticket, then reopen another similar one named "Module phase 2" or something similar for next sprint. One guy was an expert at gaming the system, and his ticket got closed and opened anew for about 3 or 4 sprints.
No one should be surprised when employees respond to incentives, and blaming them seems a clear indicator of managerial failure: failure to tend to morale, failure to reward actually useful behavior, failure to articulate a vision.
Gameable KPIs offer windows into the souls of your colleagues.
"This week we found the root cause of why operation X sometimes fails - it's a slow database query which we plan to fix next week. For those interested, here's a command-line demo of the issue."
"With 60 pull requests being submitted per week, and each one triggering a 12 minute automated test run, we were wasting a quite a bit of developer resources. This week we brought that time down to 4 minutes. For those interested, here's how we parallelised the tests."
"Remember how the last accounting feature broke a lot of different parts of the system and it took 4 extra weeks to fix? This week we migrated 4 out of 18 core accounting functions to a service completely separate from our main code base. Once the remaining 14 are moved over during the next three weeks, accounting features can be built and tested independently without affecting the main system."
So be careful what you measure and you reward.
For every company cash flow and profits are undoubtedly the most important metric. It is almost impossible to argue that maximize those numbers should NOT be a goal.
At the same time when that becomes the ONLY target that matters, the consequences are dreadful.
At least in US, that is how we ended up with appliances that only last a small fraction of time that used to last 40 years ago.
And even worse it is how we ended up with the food industry creating more and more addicting food resulting in 70% of the population be obese. And it is how we ended up with a heath system that costs multiples of what costs in any other country in the world, that, instead of healing people for good, make them "less sick" addicting them to a few pills for the rest of their life. Because there is no money to be made with a healthy person.
When you build KPIs, make sure you "think a few moves ahead" and you put other correcting metrics and checks in place. At least make sure who establishes the metrics has a way to become aware of the possible shortcomings and plan corrections in a timely manner.
Except for VC backed startups. Actually, there are a lot of exceptions. But those mostly revolve around caveats surrounding riskiness and timelines.
I am not sure this scales very well with company size. ;)
[...preamble] it was not even forgery. It was merely the substitution of one piece of nonsense for another. Most of the material that you were dealing with had no connexion with anything in the real world, not even the kind of connexion that is contained in a direct lie. Statistics were just as much a fantasy in their original version as in their rectified version. A great deal of the time you were expected to make them up out of your head."
- George Orwell, 1984.
All one knew was that every quarter astronomical numbers of features and improvements were produced on paper, while perhaps half the users of SaaS products complained or left.
When it comes time for me to move on from the company, one of my antics is going to be to reply-all to each and every one of them with the word "unsubscribe."
"What does a user see? Nothing, but the old one had bugs, including at least one CVE. Now we don't."
That seems like a reasonable content for demo and with just the right amount of self-deprecating humor it can be a slightly entertaining 2 minute presentation.
Unfortunately, too few people think that way.
It reminds me of deviantArt tech dev meetings when I worked there (~10 years ago)
Every Monday, each tech/product team did a small demo for the whole tech org (maybe 50 people, growing to nearly 100 by the time I left).
Teams without an interactive demo would just put together a web page with a few pictures and text describing what they accomplished and team lead would present it to the whole org.
Teams competed for dubious accolades like most lines of code deleted, most embarrassing feature, best meme. Prizes were arbitrarily awarded by the VP of technology in the form of credits to buy art from their prints shop.
Similarly to what you describe, this practice relied heavily on feature flags enabling us to release features for testing while they were still in early stages of development. This worked out really well for getting feedback and QA testing early on, while also keeping everyone up to date about everything that was happening outside of our immediate area of focus. It was fun and motivating as well. deviantArt did a lot of things really right though back in the early 2010s. IMO it was a really incredible engineering org and probably the best job I ever had.
Question: what the format you have used (number of words)? Kind of a release notes? Was anyone actually reading those?
"Agents can now see their up-to-date balance on the app (previously they had to wait for the weekly email). Here's a screenshot. Here's a direct link to the QA environment so you can try it out yourself."
That's usually it for a given feature.
These types of updates only work in an environment where people actively read them.
Team A: Added a wizzbazz button that says "Wizz!" when you push it with a cool noise! Look at this cool screenshot!
Team B: Worked on fixing an elusive bug causing rare data corruption, but couldn't figure out what was causing it.
Team A: Re-colored the header to the CEO's favorite color! Look at this cool screenshot!
Team B: Fixed bug causing rare data corruption. Started looking into strange performance bottlenecks in UI.
Team A: Added spiffy animations that play every time the user goes to the next page. Watch this cool video!
Team B: Improved page responsiveness issue introduced by the wizzbazz button. Look at this graph; pay attention to the ninetieth-percentile time-to-first-render (dotted blue line) for dashboard users. It does actually go down quite a bit if you look.
Team A: Added a thousand lines of code to the fluxborg module! Look at this graph! It went from FIFTY lines of code to a THOUSAND lines of code! Look at how much that is on this graph!
Team B: Removed two thousand unnecessary lines of code from the wizzbazz module. We decided not to show a graph because the line goes down and we know you'll all think that's a bad thing.
Which team is getting the axe when it comes time to lay off employees? You know. Come on. You know it's Team B. You know it's true.
And unfortunately for engineers, they have to dance to this director's negligence even if it comes at the cost of their own sanity.
The real gold is in improving what in bad need of improvement.
The companies that stay afloat are often those that are abundant in people that know what really matter and find a way to do it regardless what the middle management thinks.
Which is hard to come by, because those people aren't rewarded.
Our CEO had the philosophy that if you achieved 100% of your goals every quarter, you weren't being ambitious enough. As a result, we would stop logging progress at ~80% of our goals, which was the threshold to qualify for your bonus
However, scope on projects would change during a quarter, and sometimes the VP would become distracted by another shiny new thing and make us start on his new pet project. As a result, some OKRs were abandoned and would remain at 0% by the end of the quarter.
Now, no manager (especially an engineering manager) wants to be a bad guy with his employees, so my boss often just changed our OKRs (or, if we only achieved 50% of our KPI, would lower the KPI's target) on the last day of the quarter so that we would all qualify for a bonus.
The company was losing 18mil a year and was sold to our competitor at a 100mil loss for the parent company
I feel like you're stating this as "Look how silly KPIs are!", but your CEO is correct. The issue isn't the KPIs; it's the team members who are willing to intentionally game a system in conflict with overarching company goals.
There's nothing magical about KPIs. It's data. How you apply it matters. If your team wants to take advantage of it for selfish reasons, there's nothing stopping them.
The converse is that there are also always novel ways to differentiate yourself bottom-up from the mere shortcutters. This is basically how fads form. Something novel appears that is first a great signal for honest quality work, then it gets formalized and imitated and it becomes less predictive, so those who are in the business of creating true value must come up with new ways to demonstrate this.
One also has to consider whether the people who define the KPIs are themselves properly incentivized. How are they evaluated? Could there be some perverse incentive to set gameable KPIs that get hit easier? Maybe the KPI setter is in some sense in on the game and complicit? Very complicated stuff. What if the business gets destroyed in the process? Who does that hurt in reality and how much?
But still sabotage seems to be frequent - as part of "empire building", as part of gaming the compensation metrics. And "boys will be boys" is not the a response that steers toward the desired behavior.
And it's not just individual people modifying their behavior to game things, but more of an evolutionary process that the people who tend towards gaming things will get better bonuses, better reviews, better CVs, hence better next jobs, in perpetuity.
"Don't game it" can't be a simple solution. If there are people who game things, and they accumulate rewards for it, you will just see them out compete the nongamers. You are basically telling people to sacrifice their and their family's financial position for the sake of the corporation.
Everything can and will be gamed. But at the same time if many game it, managers will really want to see evidence of nongaming, so employees are also incentivized to signal their honesty, but this must be given some kind of channel too. If managers myopically focus on gameable metrics, then their loss. You must alwas leave a door open for receiving honesty signals through some kinds of side channels. These must always be somewhat amorphous because the moment you define them, they become gameable.
In other words, no recipe, stay alert, try to cooperate but don't be a pushover, etc.
What a cop-out. Everyone should want the company to succeed. So why not say "Hey, boss, this KPI won't work, here's what might happen..." instead of literally working less than needed to prove some point? It's dishonest.
The company has no interest in my success, why should I care if the company succeeds? It has no effect on me if it does.
"That's my only real motivation is not to be hassled, that and the fear of losing my job. But you know, Bob, that will only make someone work just hard enough not to get fired"
- Office Space
Put yourself in the same boat: if you had a team to manage, with the freedom to give them incentives, and they didn't have the effect you expected, you should be able to notice it and try something else.
That's great if the team is helpful and tells you exactly what's wrong, but you should also be able to evaluate how it's going and react accordingly, otherwise you're just asleep at the wheel.
The CEO directly created the perverse incentive: actually hitting your goals only meant more work.
> How you apply it matters.
The CEO applied KPIs in a way that disincentived hitting your targets, and the team responded accordingly.
If you are hitting your goals every quarter, that is a great thing - but I tend to agree, at the other end of the spectrum you have people setting easily achieved goals to check a box.
The real issue is weeding out people just there to check a box when you're small, and when you're large, it's designing a system that internalizes a lot of your employees are there to check a box.
More that you have to have realistic expectations when you structure an org, that people doing the work are going to pad and sandbag in successive tiers up, and people in charge of outcomes are going to push for bigger and more down.
Of course, we first would need to agree that this is desirable and that's probably the point where it breaks down.
Well the paycheck ends when the company folds. "Doing good work" is in the best interest of all parties.
So, you get another job. With great odds of increasing your salary more than you could at your current position. There's really few downsides for workers in having this attitude.
Additionally, at such a large company (1500 people), my individual efforts have little to no effect on whether or not a multinational corporation goes under.
Option A is being an average worker, not trying especially hard, not thinking about work outside of work hours, and having absolutely no emotional investment whatsoever in the wellbeing of the company - and potentially having to get a new job in a few years time, often with a pay increase
Option B is "doing good work", "going the extra mile", "being a rock star", putting in lots extra time and effort and emotional energy (for years!) - just on the tiny chance that your particular efforts will be the difference between the company folding vs. being successful
Option A sounds a hell of a lot better to me than option B.
You know what happened?
1. I got even more work. So much more that I no longer had the freedom to explore and think.
2. The carrot on the stick would be re-attached to a longer stick. It was never enough. I was performing "exceptionally" but my yearly review would never move me closer to my goal.
3. I would end up getting tied down in a bunch of stuff from (1) that ended up leading to burnout because I was no longer working in a place where I felt useful and needed.
4. I still got laid off.
Now, I just go to work and come home. I am pissed I even have to do pagerduty because 9/10 times the problem is someone wasn't given enough time, creating a bug, which was then subsequently never addressed because aGiLe methodology says you only move forward.
"High power" CEOs and PMP-certified project morons are the reason why people's care for your product ends with their paycheck. No amount of "demo days", "email updates", or metrics will fix it. I, and everyone who is like me, will game your metrics until they stop being useful. It's not malicious. It's an optimization. If you want me to do exactly what your metrics ask I will. Nothing more, nothing less. You pay me for 80 hours a check, you get 80 hours. Nothing more, nothing less. If you don't want me to think I won't. Afterall, I'd rather save my brain cycles for things I enjoy. You're paying me to squander my talent. That's YOUR problem. Not mine.
You say this as if employees are absolved of any responsibility to the performance of the company. You're not clever for saying, "Oh, 100% means we weren't ambitious? Then we'll work 80% as hard." In fact, I think you'd have to be pretty dense to not understand the intention of the KPI: set lofty goals and try to get there.
If you don't think it's working, why not go back with ideas to improve it, rather than doing less work? I mean, this company literally failed (from the sounds of it) and people here are blaming the CEOs KPIs versus the employees who intentionally work less to game the system. Strange.
I do care about my professional reputation. I'm highly skilled and understand how to improve social dynamics on a team, making me very valuable to have around. Clever leadership will figure out a way to align my skills with their own goals. But, if they don't, that's their problem.
That's the bottom line, isn't it? If the CEO wants to align employees' behavior and motivation with his own, he needs to structure the employees' compensation to be like his own compensation. If my compensation is just a "competitive" salary and a few token stonks, then I'm going to do a 9-5 job and hit the required 80% of my KPIs before I go home for the day. If my compensation consists of life-changing equity? I'll work days, nights, weekends, and holidays.
You think most 200+ Employee companies are unicorns?
As I said, I care about my professional reputation. When/if the company dies I want every single one of my former coworkers fighting to get me hired at wherever they land. You don't get former coworkers clamoring to get you in at their new shop by being a bad worker.
Companies want me to work for them, and former coworkers want me to work with them. I am just not emotionally invested in the companies who pay me a wage, I'm going to set reasonable boundaries on my time, and I'm generally going to respond rationally to incentives. Responding rationally to incentives includes considering if the extra work required to achieve a particular bonus is worth the effort, often it is not.
If you create a system which is abused by 100% of employees, that's on you. Everyone who works at a company does so for their own personal benefit. You can moralize about it if you want, but I doubt you would work for a company if it didn't provide an incentive structure of some kind.
> why not go back with ideas to improve it
I was a junior developer working for a 1500 person company. What should I do, demand an audience with the CEO, who lived in a different country?
Most people willingly turn their brain off when it comes to evaluating their daily/quarterly performance against that simple truth.
Doesn't sound like "less work", it sounds like they did the amount of work management wanted them to do. Just like working 40 hours a week is "less work" than 60 hours a week, but I'm not going to work 60 hour weeks just because of some sentimentality around the company being successful.
the description give didn't make me think it's the kind of workplace where the CEO is going around asking people, "what can I fundamentally change about our goal setting process?"
in any case, goals should be achievable, so if you punish someone for achieving them, you messed up
Tracking and measuring KPI is good, without it the company won't know it's current performance and how to fix / improve it (though beware on burnout). However when it's become target with financial incentive, it ceases to be good measure, because everyone will try to optimize things to reach the target.
Velocity (work complete / planned * 100%) should always be a measurement, because it can help to identify errors in workflow / team member, and prevent demotivation when they need to perform task outside their goal.
It boils down to that ridiculous "stretch goals shouldn't be achievable because you should be shooting for the unattainable". If you want to have that mindset, don't beat the exployees for achieving everything. Beat yourself for not curating their goals.
So, hypothetically if I busted my ass for three months straight to hit 100%, just to be told "Oh, your goals were set too low, so here's more work", what incentive at that point do I have to work even harder? The new target's 80% was my last target's 100%, which I struggled to hit (because as you said, these goals are lofty). So if I bust my ass even more, do even more overtime, what do I get then? "Your goals were set too low, so here's more work." Do I just continue this until I burn out? Or do I hit 80 of my target so my workload doesn't spiral out of control, and I get exactly the same bonus?
> I mean, this company literally failed (from the sounds of it) and people here are blaming the CEOs KPIs versus the employees who intentionally work less to game the system. Strange.
Employees gaming their KPIs is a reaction to and a direct result of the CEO's "if you hit 100%, you didn't work hard enough" policy. CEOs using constantly moving targets and increasingly unrealistic expectations to extract every last drop of productivity out of their employees is met with an equally maximizing reaction: employees find a maximum amount of time and energy they can spend such that the status quo is preserved and the bar doesn't move as high next time.
I would bet KPIs were not the main reason that company went under. Obviously we don't have specifics, and companies rarely fail for one reason. I didn't blame the CEO for the company going under either - I simply pointed out the perverse incentive.
The real issue is - KPIs are used to incentivize or PIP employees. Why are you setting the employees in a game that is in conflict with your company's overarching goals?
Upton Sinclair — 'It is difficult to get a man to understand something, when his salary depends on his not understanding it.'
I'm not sure where you see the correct part.
I have seen this over and over again - a company made up of good engineers without focus doesn't do very well, unless its has accidentally fallen in a market where they have few to no competitors and this circumstance encourages even more lack of focus, because who are you even competing against?.
Even companies that have bad practices and processes, their eng isn't that great, and often putrid management behavior, the focus on a specific goal that customers want made MONEY and sold companies.
In my experience almost all of that came down to the CEO wanting to hold the line and do the thing that we needed.
Now of course, whether OKR works depends on culture as well. Sometimes, asking people to stretch amounts to asking them to work harder, so naturally the system will be gamed in that context.
You can set up OKRs like improving customer experience and, at the end of the day, someone will declare that they have been accomplished or not based on their preference of employees or mood. That is functionally the same as saying this bonus is optional and we might give it to you if we feel like it.
It is a pity because well chosen KPIs that employees try to optimize for can make a company work great. But they need to be _very_ well chosen. This, like economics indicators, often requires creativity and technical knowledge.
As an oversight measure, they fail for the same reasons as every other oversight measure.
Hire people you can trust, give them skin in the game, and then trust them. If you exclusively interact with me via the carrot and stick, I will recognize the lack of trust and respond with an equal lack of trust.
There is too much of this on our industry, and inevitably the churn in our industry matches the service industry.
And then management is surprised when engineers burn out and stop caring. I mean, do they not understand reciprocation? Did they really think treating people like horses will keep people honest and invested in the success of the company and the team?
This is the problem: tying OKRs to bonuses. It sounds so logical from a naive standpoint and yet it has so many detrimental side-effects.
Where I'm at, we have a hybrid system that works quite well. OKRs are divorced from bonuses; and they are always "Stretch OKRs", ie. ambitious ones. Essentially they are the compass where team is headed. Then there are MBOs which are tied to bonuses. These are very conservative, so that the bonuses are attainable.
With this system, in practice, the MBOs are the subset of the Key Results that actually seem achievable within the quarter.
Of course there's a functional way to play this game and a dysfunctional one.
Working, functional version: spend time defining actual Objectives, regardless of whether we know how to measure them. That last point is very important. Only when the Objectives are defined (and they are almost always qualitative in nature), THEN come up with Key Results, to try to quantify the objective (and be open to adusting that). Finally, pick out the KRs that can be realistic, and make them the MBOs.
Dysfuntional version: start at the opposite side. Define MBOs. Then Key Results. And finally, try to BS some form of Objectives to make it all sound coherent. This biases towards both what is easy to measure and easy to quantify, to the detriment of business needs.
Goodhart's Law: "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."
Management is a service task, it does not produce value itself. Good management is important to prioritize, allocate resources and resolve conflicts. But that is not how MBAs seem themselves, they see themselves as the most important thing around, they're the ones in charge after all. And damn physics and reality, the spreadsheets say you can get get a baby out in a month if you use 9 women to do it. Soon management captures all the attention and sucks away all the money, and shortly thereafter end up with a moribund shell of the former company. You can tell how close to moribund a company is by looking at the manager to worker ratio. Get out of there once it climbs much above 1 to 10.
There are good managers around, but they're almost as rare as unicorns.
And re ex-engineer H1Bs: just because you used to be an individual contributor before becoming management does not imply you retain your individual contributor skills nor that you'll be a good manager.
Most people across my career that became managers did so because the pay was better, workload was lower and promotion ladder more straightforward. Worst managers were usually the type that expected and demanded to be management after X years as an individual contributor; this is particularly true of people from some countries, where it is culturally expected that you'll be a manager within at most 10 years of starting.
Most companies do treat moving to the management ladder as a promotion, which create all sorts of perverse incentives for many to become managers. Reminds me of a blog from a few years back:
https://charity.wtf/2020/09/06/if-management-isnt-a-promotio...
So instead, bring back the secretaries. A separate org vertical (i.e., they don't have hiring/firing power over ICs), which handles the secretarial half of your current managers' jobs, and isn't paid more than your engineers. The engineering decision-making half of the job (along with hire/fire power) should be handed over to one or two senior engineers, who are now the actually powerful ones in the engineering branch of your org chart. This keeps your technically talented people working in technical roles. Basically, instead of
C-suite C-Suite
/ \ / \
M M EM SM
/ \ / \ / \ / \
M M M M E ES S
/ \ / \ / \ / \
E EE EE EE EFurther, I also think that EMs need to then become more technical aka a real "leader" of the services they are responsible for. So hiring, firing, perf, BS meetings becomes only 5% of their job. 95% of their job is to ensure that services keep running and projects get delivered - like a true leader. ES can help them and ICs with administrative tasks like HR follow ups, OKR tracking, meeting scribing etc.
I can't believe how much work is being done by low level EMs and ICs in tech companies. It is almost exploitative in one way.
It’s pretty common ATM for product managers to not have direct programming or software backgrounds and instead have a career like “MBB consultant -> MBA -> product manager”. Big tech companies also insource a lot of “strategy consulting” which are MBA types. And marketing/biz dev have plenty of MBAs too.
While someone like a “director of engineering” probably doesn’t have an MBA, there’s a good chance their boss does (especially since these days VPs seem to mostly be former product managers rather than former software engineers).
By the way, Tim Cook, Satya Nadella, Sundar Pichai, Andrew Jassy, and Cheryl Sandberg all have American MBAs.
Scoping was whatever manager decided the right scope and timeline was.
It's almost embarrassing to see it in action. I am embarrassed that I am being asked to toil and grind for such stupidity.
Therefore, if you are unhappy with a measure, it means solely that it doesn't capture all of your preferences properly. Which is a technical problem rather than a philosophical one.
And then over time they start realizing they don’t have to lift up the whole product, but just a small piece to increase the measurement. So they do that and the measurement goes up, but the product doesn’t get better because they’ve found the path of least resistance to raising the measure. This is really the underlying crux of Goodharts Law. So it was a good measure probably until it became a target.
So what is a manager to do? “Capture all [the] preferences properly” as you put it? Probably not because that quickly devolves from measurements to long form status reports, not even measurements, because it’s impossible to capture all dimensions of this with measurements, so one has to reduce the dimensionality a little.
This is a philosophical problem not a technical one. Though your point does seem superficially correct, in practice with real teams the second and third order effects from the measure becoming a target dominate.
That assumes that the ones working towards a specific KPI are both the ones noticing the metric is bad and that their are noticing it before the negative consequences have destroyed their product / company.
Often when the issue of optimizing to KPI comes up, it seems that exterior people notice the problem but not interior ones, or otherwise the issue is noticed after the fact but not before a meaningful change could be affected.
Here, you're assuming that the space of possibilities is an ordered set. But it's probably not, so your “technical problem” is in fact a mathematical impossibility.
With KPIs, you push people to maximize a very-straightforward-but-fundamentally-flawed metric, instead of relying on their own judgment. Or when not trusting your employees end up having them behave like brainless bots.
So even a good measurement is vulnerable to this problem.
Most of the complexity, inefficiency and the related failures in today corporate and institutional world is driven by too much management. The map has become the territory.
So now line 2 is exit 0.
It's very performant.
That will be well over $20k from the corporate salary budget, thank you!
And yeah, I can totally see somebody doing it on practice. I have seen people do lots of similar things.
As a child, I worked at a large popular fast food chain. Also measured by error metrics, the trainer showed me how to fry a frozen chicken sandwich. The meat spilled to the floor on accident. He said, "here's your first lesson: no one saw that!" Into the frier they went, metrics perfect.
It's all the same bad faith. Managers put odometers on every interface, workers spend their time developing odometer foulers. I could never own a business I wasn't the sole worker at.
It's my favorite Munger quote, and it's a really good one. So often businesses invest so much time, effort, and money into everything but incentives. No matter how much you spend on culture, perks, diversity, swag etc, if the incentives are not aligned to some over-arching goal internally, you will not get the results you're looking for.
Let me ground this in a concrete example. If your KPI is a certain number of calls per day from a sales person, here's how the most Machiavellian caller will hit it:
1. Spend little time with good prospects (they're paid by call, not by conversion!)
2. Call old numbers/disconnected/rapid disconnects
3. Call multiple numbers at once, reduce call quality etc.
For any other metric you pick (conversion rate, call time, etc) there are similar schemes that have undesirable side effects. Either you very carefully build a composite framework of incentives to use, or you take the index fund approach and compensate on overall company performance (like a stock bonus, profit bonuses, etc). Personally, I think the best option is stock bonuses, but you may disagree.
Either way, getting those incentives right is absolutely key to performance.
1. I have less control over my bonus
2. My bonus depends on the performance of others
3. I can just do the bare minimum and still get a good chunk due to others
The goal is really to understand the impacts of whatever incentives you choose. I don't think there is a side-effect free incentive.
1. There is a risk that the stock loses value but stocks generally have positive EV.
2. Since stocks vest over time there is some intrinsic value in being able to simply walk away if the stock crashes (dropping your comp going forward) but being able to stay if the stock surged (greatly increasing your comp compared to your market rate). It’s kind of like having a long-dated option.
3. It generally takes months to a couple years for stocks to change in value enough for it to put you over market rate. It takes similar amounts of time for you to fully ramp up and start adding value to the company. Generally you tend to get into golden handcuff range right around the time you start settling in. Getting paid above-market in a role where you’re doing well is great.
4. You may not have enough influence to really move the stock up but you often do have enough influence to move it down (like being sloppy or doing something destructive like leaking).
5. From the employer perspective, aligning your financial interests with the board/founders/investors/execs probably means you’re unlikely to try to unionize or get angry at the company when it gives you only a 5% salary bump in a year where the stock increased 100%. Generally speaking the worker:exec relationship is less combative since there’s less of a conflict of interests. As an employee there are nice aspects of this, like the whole company having a more collegiate or team-like atmosphere.
KPI can be very useful as sanity checks and it also varies quite a bit by industry if and when it makes sense to lean more of them. Anything involving physical goods/supply chains usually benefits greatly from having good KPI in place. Same for financial information. The "I" stands for indicator, I think it's usually a good idea to set up some KPI with boundaries and alerts to resteer the ship if anything goes badly off the path (cash flow indicators for finance for example).
https://en.wikipedia.org/wiki/McNamara_fallacy
The KPI is a measurement tool not a goal.
If you run a race, your goal should not be to beat ussain bolt's record, your goal should be winning with your best possible time which could or could not be much better than usain's. It is your coach who is supposed to measure you and observe how you are running and training and help you adjust that based on his/her expertise on the subject that is supposed to use usain's record as a measuing stick as they help you adjust and adapt to break the record.
Step 1: measure only that which can be easily measured Step 2: ignore what can't be easily measured Step 3: assume that which cannot be easily measured is unimportant Step 4: assume that which cannot be easily measured is not even real
Plus yeah, also Goodwin's Law.
After all of this I once again pointed out how useless and stupid the whole process was and how easy to game it was. We could easily lie in our scripts if we wanted to, it wasn't like management was capable of checking, and it was going to encourage reclassifying/hiding certain bugs (hard issues) and pretending other things were bugs (small changes/minor feature requests) to juice the numbers.
The whole processes took a _ton_ of time and in the end I never heard another department's KPI numbers (we were told every dept was doing it and we would all report monthly for the whole company to see) and I'm fairly certain it fizzled out though I left shortly after that.
But why would you do this???
Holy cow, some of these comments. As someone fixing bugs, why would you not be interested in the classification, status and tracking of bug-related data? Why would your first instinct be to lie about what's happening with the business? To what end?
Honestly, this thread has me wondering if the average worker does so little that they see KPIs as a way of being accountable, and don't want that in any manner.
If your compensation or continued employment is tied to metrics, especially metrics that aren't inherently valuable, then there's much more incentive to game/fudge them than to do the work to actually resolve them.
We do pentesting to prevent us having vulnerabilities and maybe even be hacked.
But then the new manager wanted a KPI. In their infinite wisdom the management people decided that "Cost saved by preventing a hack, in Euros per product" was going to be the KPI.
So now a bunch of pentesters have to try and estimate what the impact of a potential, never happening (because we found it) vulnerability would have been on our company, if exploited by a malicious actor.
After a while of guesstimating this they started giving us flack for the number going down. We tried explaining over and over again that this makes no sense as a KPI. We're just guessing essentially. But no, they want the KPI and they shall get the KPI.
So now we just guesstimate the potential cost a little higher every month, but not too much.
Oh well.
Grug work hard for own KPI: amount of times not beaten with club.
KPIs (as a target) are undoubtedly good for something that's directly benefit the company (with the caveat of long or short period), and the task is static. This is good for 40 years ago where factory worker are a thing, which nowadays, those good KPIs are mostly replaced with machine specifications.
What those people's complaining about the KPIs are for knowledge tasks, usually software development, where it's full of uncertainty and almost impossible to estimate (see https://en.m.wikipedia.org/wiki/Hofstadter's_law). If you can't estimate the work, how can you make targets? You can only measure it and hope that the measurement can help for future estimation.
I don't think you can get there with KPIs, but at the same time, you can't completely discard methods like these which have some level of credibility with other stakeholders. As evidenced by many conversations on this topic involving programmers, KPIs get completely thrown out, or worse, undermined. Insufficient time and effort is given to actually making the leap to something that might be useful sustainable. Hence, many places never break away from KPIs and the complaints keep coming.
Measurement without particular targets can be quite useful when the organization is open to sharing data without punitive responses. This gives you different axes to explore with regards performance so that you can shape your messaging around performance provided you can contextualize it.
EDIT: My personal philosophy on how strategy should work and be implemented is closer to that of Mintzberg (https://en.wikipedia.org/wiki/Henry_Mintzberg) than Porter.
IMO keep measuring, identify all the problems and fix/improve them, and reward based on team achievement rather than target, it'll boost morale overall.
In my experience, data driveness quickly leads to an unhealthy focus on short term results - cause that's usually what's most straightforward to measure and impact. What you get is teams hunting local maxima. What's missing is the guts to follow a vision/intuition regardless of what the numbers say, at least some of the time.
It seems hard to accept that we can't measure everything that impacts the long term success of a company, but I don't see why that typically translates to not caring about it altogether.
KPIs might be defined incorrectly, but their use is simple and straightforward for people to understand. This gives rise to Goodhart's Law, "when a measure becomes a target, it ceases to be a good measure." The best KPI is something that can't be gamed by those whose performance is being measured. This is probably impossible in practice.
Your car dashboard is great for staying within speed limit, but now try driving without taking your eyes from the speed dial. :-)
You can fail a driving test if you spend too much time driving more than ~5 miles per hour below the speed limit in safe conditions.
The point though is that timidity in a driver is actually more dangerous than exceeding the speed limit. Being consistently below the speed limit is a strong sign of timidity.
Very common to see older people drive slow on the highway because going 100KM/h seems scary to them. This is, of course, absurd, since driving 80KM/hs on a 100KM/h road causes way more moments for bad things to happen because you're forcing everyone to merge into the speed lane, including trucks with bigger blind spots, rather then only the drivers choosing to go above the speed limit. If you do this you should absolutely be fined, and that's not because it's a good metric for timidity (which I agree can be bad for drivers in general), but rather that the action itself causes harm.
Nobody "needs" to overtake or is "forced" to doing something dangerous outside safe margins. Seriously, this is blame shifting when somebody does something stupid. When you do, it's 100% on you.
100km/h on a highway with everybody otherwise going 130km/h is rare. This is about a 65mph road (100km/h?) where somebody goes 95km/h and everybody behind them goes berserk. That's an attitude that needs to change, not by the 95km/h driver, but by everybody behind. Going 100km/h is not a right, does not force you to do stupid overtaking maneuvers and the time you lose by going 95km/h is negligible. At least it's substantially less than you'd expect, since there are many other factors that slow you down on a journey. You rarely drive for 3 hours straight, no interruptions, no other traffic on the road. Even if you did, doing 100km/h instead of 95km/h would get you there by 9 minutes earlier. That's the upper limit here, but you are likely going to save less than 5 min. After 3 hours.
You seem to have an agenda in your comments here, and in some of them I think you have a point, in this one you are overplaying your hand.
I'm a bit confused regarding your insinuation of an "agenda" in my comments? I've been in enough situations where somebody behind me or others on the road was endangering everybody just because the car in front of them wasn't going the $safe_margin_above that they were expecting of everybody. So they go like 3ft behind them and put pressure and stress on people. It's a dangerous mentality that can get people killed. It's not the "slow person" (going at 63mph) that's the danger in that scenario.
My driving instructor was (rightly) very hot on the principle that you must always be able to stop within the distance you can see. Speed limits are not set with this in mind.
In the UK, for example, that means that on winding country roads you should often be doing a lot less than the 60mph speed limit.
Nearly every production vehicle available today can brake from 60mph in less than 140ft, more performance oriented vehicles can do so in under 120ft. The standards for “safe stopping distance” used in the US triple this distance to account for slowed reaction time because drivers are not attentive.
Realistically less than doubling these figures is correct for attentive drivers. 60mph is 88 feet per second, and 1 second is plenty of reaction time for an attentive driver, meaning 200-240ft total stopping distance, closer to double the distance the vehicle is capable of rather than triple.
So if the visibility is less than 200-240ft, you should be going slower than 60mph. 240ft isn’t very far in a car though, it’s roughly the distance traveled in 3 seconds at 60mph. There’s certainly some country roads I’d slow down on, but if you’re paying attention you’re well good on most.
For what it’s worth, the average reaction time of a gamer is 150ms, so a full second is plenty of margin of error.
It's rare that those gamers have a crying toddler in the backseat while being sleep deprived and the sun coming from a less than perfect angle.
This is a bizarre claim, since most accidents/deaths would be avoided by cars being slower. Unless you have data on this, this is just "I like to drive faster" with more words.
At any rate, neither statistic has anything to do with my claim. Timidity is dangerous because it’s unpredictable and creates a rolling roadblock. The most dangerous thing while driving is /other drivers/. Being able to predict their behavior and avoiding having to overtake both greatly reduce risk. Timidity increases both of these risks significantly.
Citations definitely needed. Not just of the studies that perhaps support your point, but of the reviews that do a quantitative overview of that the majority of studies conclude what you claim. Everything else is "I want to drive fast, slow grannies out of my way".
Research is clear. For every 5mph increase on a highway you can calculate how many people per year per mile will die more.
If you want more people to die, then sure, speed limits are too low.
Could you point to your research.
There are situations where it doesn't. These are mostly also situations where the speed limit doesn't work terribly well as a limit either. For instance, bad weather or winding roads may mean that driving at or near the limit is unsafe. In that case, sure, it's not good for everyone to coordinate on driving at the speed limit, but it's also not good for everyone to think of the speed limit as the right upper bound for driving to be safe.
So I think that in most situations either (1) treating the speed limit as a rough target as well as a limit is mostly a good idea, or (2) the speed you try to use as a limit should be something other than the nominal speed limit.
If driving at the speed limit on a winding road is unsafe, the speed limit is probably too high for that road. But in practice, it's complex, winding roads that cause people to drive at reasonable speeds, rather than numbers posted on signs.
Management want indicators of certainty and progress with the least effort possible. Superficially, dashboards composed of KPIs give them what they want. Good CEOs are unlikely to rely on these, but instead think of them as an alignment and investigative tool.
Each layer of the organisation aligns itself around some KPIs with lower levels often feeding upwards through the hierarchy. As you further down the hierarchy things get messier as they are further from the executives, may have less experience/skill with using tools like KPIs in the way the CEO wants, and there many more teams to align across with problems like duplication, etc.
The majority of engineers are at the bottom of the pyramid. Their management is coming from their own ranks with little support, so it's not a surprise there is misunderstanding and misuse. In programming, when the career ladder often involves stopping dev work and becoming a manager, it only makes things worse. When I see peers promoted into management, I always hope it's the ones that have capacity for leadership versus experience because they are going to be the ones that adapt KPIs to match the internal and external needs of the company.
In my experience it's not data, but when finance and/or sales call the shots. Their incentives are quarterly so without good engineering pushback you get a lack of long term vision
Also, because defenders of KPI-driven management love to use the saying that "you can't manage what you can't measure", it's worth repeating the original quote where this is lifted from:
> "It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth." (W. Edwards Deming)
The problem is the middle manager owes the VP 12 measurements. The clever supervisor picks the easiest to achieve metrics possible.
Ultimately, the "some level" you refer to is a very basic level; even more basic than what one would intuitively think.
They brought with them the church of Jack Welch, which was ironic as that culture pretty much chewed them up and spat them all out.
If the VP measures “goal accomplished” instead of “potential goal impact * % realized” then the VP will get easily accomplished worthless metrics.
The dashboard was 100% green - we tried to do x, y, z. The auditors were laughing when they verified our attempts. But the bonuses were paid and all was well.
So instead some second-in-command on the team has to come up with some KPIs that they will have no control over addressing (they aren't team lead, they aren't product, they don't own the backlog or roadmap).
Inevitably there is a bunch of angst from management that the KPIs are too tech/engineering centric (well..) while again providing no actual good KPIs themselves, and then they all get missed anyway since no one is incentivized to address them.
One of my KPIs was about selling more headcount to the client.
None of the KPIs that came down from on high made any sense. I could be the best junior engineer in the company and it would have no impact on my performance review.
"I'll know it when I see it" iterations of bringing different "no not that" KPIs to management is frustrating in the opposite direction.
Ultimately management wants to set the tone in both scenarios, at least when giving you dumb KPIs they have done some work and volunteered information.
I worked for a company that wanted some growth in business, so they started to sign contracts for jobs for which we had little to no qualification. We got new customers, more business, more work, but employees started to leave or burnout.
There is a saying in BI that goes: “What you measure grows.”
So let’s say you measure MAU, the infamous vanity metric. And let’s say your MAU increases with each dollar you funnel toward advertising, the thing you’re actually growing there is your dollar spend and not the growth of the user base.
Thus it is your advertising dollar spend that will grow and not the size of your user base.
Every year some new hire would go out and sale 15% of sales plan and would be praise as a superstar. The next year they'd be written up for some bs reason and after a few years they would be flushed out.
What did they do wrong? They out performed sales plan in the first year by 15% and that meant that now every other year their plan for the year would add 5% growth on top of the 15%. Since the initial 15% growth was unsustainable the waste grew and magically they were now failing at the 2 major KPI's.
Currently two of my OKRs for this quarter based on what I thought a project was going to be and have since discovered I was wrong, so they don't actually matter and I'll have limited opportunity to achieve them. Great, so now they get a fun data point can point to in order to give me a smaller bonus next year.
> The McNamara fallacy (also known as the quantitative fallacy),[1] named for Robert McNamara, the US Secretary of Defense from 1961 to 1968, involves making a decision based solely on quantitative observations (or metrics) and ignoring all others. The reason given is often that these other observations cannot be proven.
> US Air Force Brigadier General Edward Lansdale reportedly told McNamara,[4] who was trying to develop a list of metrics to allow him to scientifically follow the progress of the war, that he was not considering the feelings of the common rural Vietnamese people. McNamara wrote it down on his list in pencil, then erased it and told Lansdale that he could not measure it, so it must not be important.
Time and time again I've seen managers come up with a Great Idea(tm). It works for a little while. And then it turns into a shitshow because it goes from being a useful idea to being dogma.
Agile has automated micromanagement for management ...
For the life of me I couldn't really understand why 9 months ago when a new CEO came to our company and first thing he said "measure everything, every team should have public OKRs", everyone from my team jumped on that OKR/"metrics" train and was hyped to have it. I was the only one against it, but that didn't make any difference.
Two quarters later it was pretty obvious nobody gave a shit about those (we had much more important stuff to do) but our manager had to apologize to upper management quarterly for "poor performance" of our team. And informally, nobody was taking any of those OKRs seriously, and colleagues thought of it as a bad joke.
I have no idea why the same team members didn't voice their opinion immediately and shot that idea down while it didn't take root yet.
Now half of the company is getting laid off behind a thinly veiled excuse of a RTO (from a company which was remote first since the beginning, and much longer than covid years) and everyone is a surprised pikachu. I'm not saying this is solely because of the OKRs, there were more red flags besides that one. I'm actually surprised that very few saw it coming. That the CEO is trying to emulate Musk and do a Twitter speedrun, but with someone else's money. And blame engineers who actually made the company as known as it is.
(Yes, much of the standard model and modern computational science muddies this point. That’s illustrative and worth contemplation. Obviously, such are only as good as their data and calculations agree with reality.)
1)Lack of clear mission/direction at the top level of the organization. This is a 100% necessary condition for good goals & kpis throughout the rest of the org. Otherwise it feels like political battles and favoritism between different people or departments with conflicting goals.
2) Goals & KPIs that are not well mapped from the org's mission or roadmap down to the individuals. Now all the individuals can hit their goals but org fails, or vice versa. This feels like "doing what's right for the company is bad for my performance review."
3) Confusing health KPIs with outcomes or goals. Something like commits per day/month is a great health check, but bad goal, and the benchmark for healthy is totally context dependent. This feels like "She solves the hardest problems but the solutions are very few LOC." Or "he has the most commits because his code is so buggy".
If they were just called metrics in most companies and not Key Performance Indicators (KPI), then they would get less focus. KPIs has been a heavy consultant pushed term but they are just metrics, they should never been seen as how to create value and develop a product.
People love to say "KPI" as much as "Agile" and "velocity" nowadays, both have been horribly twisted and ruin agility and value creation over value extraction. They need to be generic again... metrics, agility, delivery and usability.
When metrics are all you measure, you end up with problems [1]
[1] https://en.wikipedia.org/wiki/Performance_indicator#Problems
Later I found out this rule was articulated in mid-XX-century in sociology -- the Goodhart's law: "a metric stops being a good metric as soon as it is used to make decisions".
"How long a PR stays open" was one of the KPI in a dashboard.
I said: "Be careful with that!"
People started to close PRs and reopen new PRs with the same code.
Middle managers and sometimes the person in the division that was the point of contact for the velocity initiative were asking to do that.
The script measuring this KPI was improved to look at the branch and the diff of the code. Result? people changing branch and EOL encoding in the new PR
Learnings? B and C players with questionable ethics screw companies quite rapidly.
If you do not have an outstanding company culture, KPI and aligning them with company values is futile.
Here is a book that makes that quite clear. Adam Smith published it in 1776. This is not a new idea. https://www.gutenberg.org/ebooks/3300
I remember a recent story about Ziff Davis killing old web pages. If that is KPI psychosis, the fault lies with the person who set the KPI to be average page hits, not total page hits across their web footprint.
At the same time, they forget that not everyone, not every team is working on the same thing. So the KPI (whatever it is defined to be) is not measuring anything of value.
But a manager will never admit that they are measuring the wrong things. So engineers are stuck with poor practices. Go improve your number of commits even if is meaningless.
Except what happens is either engineers game it by just canceling PRs, making the requested change and resubmitting it. Or fighting tooth and nail for everything. No I won't change the naming on this variable because then I might get out on PIP. Or even more insidious, the team just doesn't do real code reviews, they just rubber stamp each others work in a "you scratch my back, I'll scratch yours"
If it were not for the unrealistic anti-competitive moats of these companies, these management practices would have burned them to the ground. Startups that have copied these practices without giving FAANG compensation have faded away in history.
But hey, its not my company. If they want employees to spend time on BS, I'll play along as long as I keep getting my money. I don't care for the product, services or the company at that point.
But ranking and PIP? I assure you you are not incentivizing collaboration. You can try to force collaboration but most employees will do just enough because it is in their best interest if someone else fails.
If there were a system design interview where you had to design a system in which employees throw others under the bus - but subtly - what would you design?
Have you ever talked to an engineer that was forced to work with someone on their team who providing negative value? That doesn't drive collaboration, that drives low morale and eventually people moving on. If you need to identify low performers then you now have an metric explicit or implicit that identifies them.
Have you ever worked on a team where people are yanked every 6 months systematically? No one on such teams wants to collaborate. It is in an individual's best interest to let others sink. Product, services, outcomes be damned.
It's mind-boggling that people who are specifically selected for intelligence are subsequently assumed to be complete idiots. This is why your business depends crucially on one bit of software that only one person actually understands.
An example: I live in Czechia and our universities are mostly publicly funded. In an ideal scenario resources should be allocated to achieve some noble but vague goal - to have a have population as well educated as possible, while also doing it efficiently in terms of money and time. But how should we divide the money available between different universities to achieve this goal?
One of the most important factors currently used is the number of students studying at each university. But this indicator leads to weird behavior. The universities start to favor quantity over quality. They might, for example make the exams easier to keep weaker students around. Or they might admit more students than teachers can reasonably handle.
You could choose a different metric and solve this issue, but most likely some other damaging behavior will replace the previous one. At the same time, relying on intuition might not play well with the current push for governmental transparency. It would require an enlightened incorruptible leader to make the decisions. And the public might demand objectivity.
It's a difficult problem.
This was a startup that was struggling to find product-market fit. Things were generally disorganized and there wasn't a lot of discipline. I couldn't get the data scientists to use the same version of Python or use version control in a disciplined way. The engineering manager told me about a large number of practices that we allegedly used in our code and also told me with did code reviews but when I got involved with that code I found those practices were not being applied and I can't imagine we were really doing code reviews or we would have known.
The most challenging thing I noticed was that we were doing a lot of zigging and zagging to keep up with our enterprise customers who had very different projects. Some of that was intrinsic to what we were doing but it also meant we'd spend a lot of time developing something and then abandon the work which had us spinning our wheels. What we did need, in my mind, was the discipline to deal with that situation, when I am working on my own things like that don't bother me much because I take good news and usually do a good job of restarting projects that are interrupted but my team was not so good at it.
My impression was that the investors really liked the CEO and believed in his vision but didn't trust him 100% so they released the money in dribs and drabs which didn't really help.
We had some consultants tell us all to do OKRs and we were all expected to make up 10-20 quantifiable goals and it struck me (1) a personal distraction, (2) a game that a narcissist can play to make management think his glass is half full and yours is half empty, (3) a distraction for management from "the goal" of finding product-market fit.
A national olympic team (or any competitive sports team) using KPIs probably would not experience these problems. They might get over-exuberant and over optimise slightly. Athletes might game the system slightly. But, it's unlikely that a professional sports team degenerates to levels of KPI affliction like an average corporation.
No matter the corruption, olympic teams are made up of people who really want to win medals. Athletes, Coaches. Dieticians. Managers. Administrators. Etc. They're not angels, they're just not as focused on appeasing their boss, managing up, and such.
Group size plays a role. C-level distortion fields play a role. The orgs age. etc. Legibility. "Market" feedback loops. It all ads up to an organisation that can take any management tool and make it pathological.
A commercial example of resilience to bullshit is a classic factory, that ideal "Firm" we based out economic models on. "The job" of manufacturing objects is legible, can be effectively reduced to component parts, measured and optimised. The feedback loop any significant successes and failures is effective.
A young startup with a "lets make it work together and we'll all be rich" vibe can probably use KPIs effectively. There are lots of example, and we intuitively know how this is going to go.
I think it's time to look at some microeconomic questions with the reverse of conventional logic. Why does the financial services sector have sop many employees? Not because of the marginal profit value of the least employee. Because of the revenue. A company with $Xbn and a fat profit margin is (often) going to employ lots of people because it can. At this point, all you need is to lower efficiency to achieve the required output.
Yes, the team can make the playoffs, but ultimately baseball is an entertainment product -- the goal should be for the games to be fun to watch!
And to empower your engineers to achieve that, you should
- Train them so they gain specific knowledge on whatever tech they are going to use. Most likely one of the big three cloud providers. Some companies avoid this step because they want to invest the time and money required. Often with severe consequences when engineers deliver a botched solution.
- Make it a realistic goal e.g. you may need to create a new team with a smaller scope, so those people are efficient and not distracted by other stuff. If Mary from accounts is pinging your engineers about some legacy system that keeps failing, do not be shocked when "your estimates were wildly inaccurate".
There exists a practice called OKR. Intel has been using it since the early 70s, Google since year 2, and pretty much everyone in SV has switched over since around the time that John Doerr book came out. It’s designed to solve exactly all the problems of KPI. You shouldn’t spend much time figuring out what to measure and whether you are measuring the right thing or how to put what you measured into context. You should be defining simple objectives and key results what will give you a binary answer at regular intervals. Practicing KPI solely is putting the cart in front of the horse. You should know your objectives and milestones before you figure out how to measure them.
> set stretch goals so ambitious, even if you game the completion percentage, the simple answer to whether you are doing the right things or performing should still be immediately answerable.
"John your commit frequency dropped 25%, are you quiet quitting?" "Sorry boss, I was on sick leave. Next time I'll be more active."
If we had the next box of the comic it would say:
"You're good John, you don't have to be more active, and I'm glad you are feeling better. If you need anything let me know."
The KPI is a leading indicator of something potentially going wrong (or right!) and a signal for management to look deeper. A good manager would connect with the human-being behind the metric, get the full picture, then help.
If management is treating a KPI as a source of truth and ignoring the complex people behind them, then it's a manager problem.
KPIs are not destroying business, bad managers are.
---
Boss: John, your commit frequency dropped 25% last week. Are you quiet quitting?
John: Sorry boss, I was on sick leave. I'll be more active.
---
Before you cry "this is a contrived example!" Look at what Amazon just did. They sent out an email threatening folks who weren't in the office 3 times a week in the previous two weeks. I did not take into account sick leave, vacation, FMLA, corporate travel, anything.
You take a vacation? You and your manager got a threatening email about your KPI you're not meeting.
He was absolutely miserable there.
I'd understand "time-to-first-byte" as something that could at least give you a reliable lower bound for how long users have to wait, but I don't see how "time the initial response has been received" is any kind of meaningful indicator of UX in modern javascript-driven websites.
Let's say you have a goal to increase revenue by 50%. You come up with a plan, and assign different sub-goals to departments like Marketing, Sales, etc. Each person then gets a sub-goal down to the lowest levels of the org.
KPIs provide incredible org-wide alignment which no other management technique before, or after, them produces nearly as well.
Balanced Scorecards provide a holistic view of how different aspects of a company's activities relate to each other and to the overall strategy. They help identify causal relationships between different indicators and goals. It's important for leaders not only to define KPIs but also to clearly explain the logic behind their formation to KPI owners, and to demonstrate how these metrics fit into the context of the strategy and the BSC.
Unfortunately, a lack of information and understanding of this logic can lead to misinterpreting KPIs and, consequently, to misaligned behavior or a focus on irrelevant metrics. Therefore, it's crucial for leadership to actively educate KPI owners and help them see the bigger picture — how their work contributes to the achievement of the company's overall objectives.
You absolutely can get great outcomes without putting KPIs or metrics on a pedestal. And for sure, metrics should work for the employees; employees shouldn’t work for the metrics.
The problem is that this kind of falls apart in large organizations because, from the perspective of CEOs, execs, and upper management, leaving wiggle room requires placing a high degree of trust in line teams or unscaleable micromanagement. And you can argue that the high degree of trust should be given, but bluntly, unless your employees are very well coordinated and your management team is really on top of their game, a lot of teams will flounder without clear goals.
Now as an IC that sounds like bullshit, but let me ask you: have you ever seen a team that seems like they basically didn’t do anything? What did that team have in commmon? Usually it’s at least two of 1. a lack of clear goals 2. low to average levels of motivation and personal ownership 3. low to average levels of skill. If you’re on this website discussing software and KPIs in your free time you’re probably above average in motivation and skill. But I’m sure you know that without clear goals, people just working to put food on the table tend not to have the business interest and drive to create actually-valuable work.
If you can keep your business small enough that pockets of low-productivity can’t fly under the radar, or somehow hire only motivated, skilled people that stay motivated and skilled years after they’re hired, you probably do need thing like KPIs to hold teams accountable. Not your team necessarily, but if say 10-20% of teams will flounder without KPIs, it’s probably just good policy to have it, even when it annoys the other 80% of teams, as long as you don’t add pathological processes around the KPIs.
“Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.”
https://quoteinvestigator.com/2010/05/26/everything-counts-e...
KPIs are fine when treated as health indicators. You can deliver a feature, and then watch over a good period of time whether the health improves. This "good period of time" can be months, or for something that's intuitive – e.g. you're launching a feature to be on parity or enter a new market or catch up with a competitor – this can take YEARS for your customers to understand and move closer to your product.
What backfires is, using KPIs as quarterly goals. Imagine this, you launch a feature and expect within the quarter that this feature is going to grab X% of your competitor market share (beautifully expressed as "increases X business by Y%"). This is where it starts to fall apart. Things that are human intuitive for the product (common sense market features, foundational work, new market launches) – will never get prioritized because they won't move a number in that quarter.
Using KPIs as development cycle goals are destroying businesses. KPIs are fine as health indicators.
The problem is, using KPIs as health indicators – works only with the kind of environment where: Teams after healthy discussions can trust and commit to delivering features – and somebody takes accountability for the KPIs asynchronously behind the scenes. Where a shepherd steps up and says, "I approved the features last year, I take responsibility if they all didn't change our trajectory, some did, some didn't". Today's adoption of KPIs is an "empowerment" trend, where managers figured out – teams are delivering features and I'm answering KPIs, why do this roundabout, let me just pass on the entire KPIs to the team, problem solved! "I forwarded the KPIs to my team and they committed to doing these features, I did my job, the team is not performing". The path to hell is paved with good intentions, but there are also nefarious habits.
https://www.mrowe.co.za/blog/2019/11/when-a-metric-becomes-a...
HR KPI — hire 20 engineers: you can hire 20 morons.
Sales KPI — close 100k of business: you sold the wrong use case and all of those customers churn
Eng KPI — reduce bug frequency: you make reporting bugs too onerous to do
Bad execution isn’t fixed by KPIs, it’s fixed by good execution. Good execution benefits from KPIs.
Full disclosure, it is my own rant about performance management.
The development of the spreadsheet was the beginning of the end.
This leads to perverse incentives where what the organization's leadership really wanted is set aside in favor of maximizing scores by whatever means necessary:
https://courses.cs.duke.edu/fall01/cps108/code/jpixmap/image...
Like any system, KPIs can be tweaked and made to work for you, to accomplish your goals. The key is to start by asking why you want the metric.
Sometimes the answer “to appease management” is the right answer. Marketing does matter! More often though you can come up with a better goal for your metrics and it will make your team and company more successful.
Things that make tolerable broad status indicators make horrible KPIs for that reason.
This actually give KPI some weight, because it literally defines what you do. Whenever you define you KPI, you'd think, "okay, is this all that I'm doing now?". And whenever you change your KPI, you'd be thinking, "okay, should I abandon the last game I played and play a new one?".
And that's where everything goes from wibbly-wobbly to "I've fallen and I can't get up!"
What is "human intuition?" Do we all have it? Do we have the same "it" or is this just something we accuse other people of not having but that we can't quite define, like "common sense?"
This statement may as well be "ideally, we should use KPIs with ample hand-waving, for some unnamed value of 'ample.'"
There are multiple reasons why Goodhart's law tends to be true, but the most obvious one is as soon as people understand X is important, they do or inflate X. If they successful do X, it's often at the cost of other equally or more important but untracked activities, if they inflate it, it's obvious just as bad.
I pretty quickly realized that their outages were due to their KPIs being based on jira and GitHub statistics only.
The hilarious thing was that after 6 months they started to complain about my GitHub stats. And everytime I asked why they are not tracking downtime since that is why they hired me in the first place. And they basically said it was too hard to track.
It’ll never be perfect, but it’s probably better than nothing. Company-destructive assholes and metric-gamers can run their playbooks with or without KPIs or OKR:
“Ha, you want a brake and a steering wheel? Those are annoying and distracting, do we push the gas pedal or not?”
Yeah it’s complicated but all these components are necessary.
It takes a certain amount of rigor and product maturity to make metric-based decisions work — and some types of applications are better suited to this than others.
I wonder what's the reason for this clickbait title?
Key Performance Indicators
Here is a scenario:
It's January and Sales is getting feedback from customers that if the business has a managed k8s offering, they would not only buy it but expand their use of the platform generally. Sales passes this information to product and product does some research, finding that indeed there would be a market for a managed k8s offering. They pull together a report and start to shop it around to get feedback and buy in from around the company, consulting with different orgs. After a while, a business case is flushed out and the management green light an MVP k8s offering that will generate revenue by the end of the year. An objective and key result for the business becomes: this year launch a managed k8s offering so a small number of customers with an eye for $50,000 MRR.
Every org should then have some performance indicators that indicate that the business is tracking towards the release of this product this year, it doesn't not need to be complex just as the OKR should not and does not need to be complex, it's simply an indicator that we're on track (and it's ok if you're not, maybe the OKR was unrealistic, that's ok!)
Eng: KPI: stand up k8s - KPI: make k8s stable - KPI: make k8s more stable - KPI: tie k8s into the rest of the stack - etc
Product: KPI: Research how k8s should be be exposed - KPI: Research how k8s service should be billed - KPI: MVP product design and test internally - etc
Support: KPI: understand staff to support a new offering - KPI: Training and hiring for new staff to support the offering - etc
Marketing: KPI: understand the message to the market - KPI: understand how competitors are marketing their k8s offering - KPI: understand how to advertise new offering - etc
Etc - etc.
OKRs and KPIs do not have to be a whole song and dance, they are simply a way to keep a company of organizations aligned and moving in the same direction regarding agreed upon objectives for the business, folks get super hung up on granularity. Doing this well helps tie together some of the most important aspects of a healthy and successful go to market: Vision, Mission and Purpose. I understand the vision of the business, the things we will make true in the future. I understand the mission of my team and organization on how we will realize this vision. I understand my purpose within the vision and mission and why I am an important component.
I firmly believe it's only with this as part of your foundation can achieve employee satisfaction, have directionally forward teams, and become a "high performing business".
why do so many people think that acronyms used in their bubble will be understood more widely. take IC for instance.
These are objectives, not performance indicators. Please do not redefine KPI to roughly have the same precision as "cloud" or "engagement".
Please don't redefine KPIs to trend with your experience at small business that will never scale to public ones. Your comment is exactly why business fail.
Here are some marketing KPIs:
- New customer acquisition
- Customer acquisition cost (CAC)
- Average deal size
You'll notice that "did some event" is not a KPI nor is "learned about the competition" - at least at most companies I've worked for, started or been employed by.
You've been a part of very good teams, and held some important positions. So have a lot of people on HN,and it really undermines what you are trying to communicate when you throw around "you don't scale" or "you are small business". You should assume you are among peers here, and argue on the merits, not by throwing shade.
Also its unclear how the team KPIs tie into overall Objective (Key Results are missing here). The way its structured, every team could meet their KPIs and the company could still miss the objective. Eng can build a stable k8s, product could research all the things they want, support could understand everything, marketing could understand everything. If customers don't buy the product, the objective has been missed
I don't say this to pick on you but these examples are typical of what i've seen in practice when orgs roll this out. They end up not really groking the idea behind OKR, and slap the label on a process that is largely similar to what they had before but with a trendy new name. The net result isn't better business outcomes but instead a lot of toil, infighting about why OKRs were missed, gamed KPIs, vague and unhelpful "targets", and a distaste for the whole process.
If the goal is to build a managed k8s service, a team of talented people can build it without the need for OKRs.
Coming from an SRE backround I see a better OKR breakdown on the Engineering side as follows:
* Objective: A stable Managed service that the business can sell * Key Result: - A monthly availability of 99.99% * Key Result: - Monthly P90 API latency below 500ms * Key Result: - a new instance of the service serving customers in the new region by the end of the quarter.
(Yes, these are really just SLOs, they're a good fit for this)
I know the objective rolls up to the business goal of a revenue target. I know that if i'm meeting my own goal buy the metrics we can measure from the service. I know how to prioritize engineering work. If one target is being missed consistently, the fixes are going to be prioritized At review time, I can point to improvement work that fixed/improved the results an know the business thinks its important.
Give me some vague objective and unmeasurable of researching and understanding something? I'll BS my way through filling out the form and go back to improving the service
Also, when KPIs were hit in a time frame, who talks about 20% of certain teams not being there anymore? I do agree past performance is not a guarantee. It’s kind of a dirty secret that yes this sales document is truthful but it represents conditions not current.
What is really destroying business in US corporations is woeful understaffing in “gets things done” positions. Mostly due to Wall St metrics. Book the sale then revenue and bonus…then get out the kitty litter. It’s been a race to the bottom for 10 years in professional services and lower tier positions by the dozen.
Of course the new fad of “AI will make this more efficient with fewer people!” appeals more than “wow we need bodies to move figurative data boxes around…why don’t they want shit wages limited healthcare and terrible hours?” when talking to investors.
This is simply a problem of engineering capital to Human Resources but the invisible hand has been cuffed for so long there really is no middle class anymore. The rich don’t spend. The poor spend all they have and because milk is fucking $5 a gallon, gas is $3 here in Texas, and bills ain’t getting cheaper during 112 degree days - wanna know why the economy sucks?
When in lockdown basic income to not die or be evicted didn’t ruin society. Now that we’re out of the pandemic it’s forgotten. I’m sure lobbyists will somehow change the lesson to be “we need to buy guns from companies and give them out free that will solve poverty” and Ted Cruz will be right there eating bacon off their…
Businesses are self destructing. Not my monkey not my circus. Cassandra out.