Hacker News Reader

Comment by sriku | Hacker News Reader

Policy formulators really need to understand Goodhart's law - https://en.wikipedia.org/wiki/Goodhart%27s_law

- If your "productivity" is measured and rewarded based on lines of code, you know what'll happen.

- If a QA engineer is incentivised based on number of detected bugs, you know what'll happen - a lot of trivial bugs gets reported.

- If a dev is incentivised based on number of bugs per line of code, fewer bugs will get reported.

... and that's just w.r.t. software metrics. At the country level, we have the GDP to thank for if a govt targets its policies towards influencing the GDP number upwards to the detriment of what the people and economy really need in their context.

View on HN

Perhaps a more general observation is that a measure that faces pressure by intelligent agents needs to be monitored and adjusted by equally intelligent agents to patch the exploits.

So you can either use a simple measure that doesn't work, or a complex measure that does work, but at potentially great cost (evaluating all the terms, bureaucratic inertia by the controllers, etc.)

Some metrics seem to provide more of a free lunch - be more robust - than others. It's not obvious which are robust, however.

This is everything in society, and many problems arise because those manipulating the measure have more to gain by challenging it, while the checks and balances only face loss.

Sounds like this would just lead to an arms race between smart managers and smart employees.

The best model for a company is based off it's performance as a product. If your products are doing well then the team gets more funding. If not doing well less funding and cuts would have to be provisioned from that.

Performance is messaured in five points. Sales, adoption, support, dependency and need.

Sales, how much money am I making.

Adoption, how many users are joining.

Support, how many people current actively use the product.

Dependency, how dependent are it's users on the product.

Need, how much we really want this product to be successful. Some products purely exist on a need basis.

But sometimes, the reason you are failing is because you are missing a key person or lacking funding, so it isn't quite as easy as that (and this is even said as someone who thinks most software companies have 10-100x too many employees ;P).

Honestly, I think the real problem is blindly following "metrics" of any form: metrics are input to a decision making process, but can almost never be directly optimized without causing some kind of serious mistake (which is sad, as humans are often biased; the real solution is thereby to not build large companies, lol).

I've seen the Goodhart'ing of this product orientation lead to bizarre product segmentation detrimental to positioning the company within the actual market. People keep trying to come up with a way to "systematize management" into synthesizing leadership, and I keep maintaining there is no substitute for actual leadership, which is as much an art form as non-trivial debugging and coding (especially at scale and speed).

Aligning incentives would be so easy.

And yet profit sharing is so rare.

After all this metric stuff, I’m much more in line with the idea of embracing the difficulty of assessment. What are the exact metrics I need to get promoted? There are none. I have heuristics for when it’s beneficial to reward an employee and increase their influence. Even I don’t fully understand them. Here’s some vague hints. Meet those, and you’ll be promoted.

At least it’s honest.

It is also correct. Eventually you will have someone great who fails to meet one metric, but you need to promote them before they decide they aren't going to be rewarded and leave.

Which is really what this is all about: ensuring people who you want to keep stick around. The pay you give someone is in part a reflection on how willing you are to leave. If my boss reduced my wage to minimum wage I wouldn't quit today, but you can be sure my top priority would be finding a new job. I know it is possible for me to feed my family on minimum wage, but I wouldn't like the lifestyle that means and I wouldn't like the snub.

Yup. The thing that's disturbing about this is that someone senior must understand Goodhart's law in the company. Which makes it more likely the side-effects are, in fact, intended.

Plenty of managers know about Goodhart's Law - they just also know about the power of incentives / KPIs / OKRs when properly managed. And they imagine they'll properly manage them.

After all, you can't believe in perverse incentives without first believing in incentives!

People in business largely believe in sales people getting bonuses for closing big sales, CEOs getting bonuses for raising the stock price, and workers getting bonuses for delivering projects on time and under budget.

At the end of the day “someone senior” is just protecting their own neck. These people got there for being “a team player”, ie. They play the game, in the office, in emails, in meetings, at parties and many in their own heads since school age.

At the end of the day they couldn’t give a shit about Goodhart’s law and all the book they were made to read in order to keep playing the game.

The thing is, unless it comes all the way from the top, it’s not generally a good idea to disagree too vehemently with your boss.

Especially if he thinks stack ranking is the shit.

“All my life, I’ve been in the top 5% of people who understand incentives, and all my life, I’ve underestimated it.” — Charlie Munger

Quoting from memory. It’s true that almost every time I see a really strange group behavior, it’s the natural result of poorly conceived incentives.

This is a problem that is likely to be with us forever.

I think this is not correct because many of our biggest and most successful companies stack rank. If there is a substantial productivity advantage to not doing it then the next company with a million person workforce has some low hanging fruit to pick.

Like an evolutionary process, companies are only ever graded on the sum of their policies. There is almost never a company that offers almost everything the same but no stack ranking; it would not surprise me to see that the market for labor is “inefficient” here.

I wonder if there are management studies on introducing stack ranking to a unit inside of a company?

If there are, they are not framed in societal terms but in Darwinist terms.

In regular society, it's more profitable to be a criminal/mugger/what have you, viewed entirely in a practical sense. It only becomes unprofitable when (a) there's law to prevent you and (b) there's a general idea that preventing you from doing that is correct in some way.

If anybody's experimented with this, and I wouldn't be surprised if Amazon had as it seems like they'd try anything to see if it worked, I would guess that at scale stack ranking produces slightly more efficiency and productivity than it costs in harm to the well-being of workers. Since Amazon doesn't care at all about the well-being of workers OR employers… they're like a sort of pyre to the cause of scale and profitability… then stack ranking would stay.

If it didn't return some arbitrarily small (or large, as far as we know) benefit to the company at the cost of the well-being of the workers, it would be gone. Somebody's got to have measured this by now, and all these tech unicorns are smart enough to ditch it if it was actually impinging on their productivity. It's not, it's impinging on worker conditions, so win/win as far as the tech giants are concerned.

It's hard finding counter-examples since no two companies are in the same business and situation, but consider Microsoft: stack-ranking coincided with their lost decade where they stagnated, ceded control of mobile & web to Apple & Google, etc.

A new CEO changed many things so it's hard to say whether stack-ranking was more a cause or symptom of Ballmer's bad management but from the outside it certainly seems like a healthier company now that it's gone.

That's also important to think about from the other perspective: stack-ranking seems like a frictional cost — Microsoft would have been profitable no matter what since anything short of discontinuing Windows/Office would have left a massive cash flow, but it's harder to see where pushing people out cost you until after the fact. Amazon has enjoyed a massive first mover advantage from the cloud and they're certainly executing well now but I would not be surprised if, say, strong competition lead to a point in a few years where they might wish that they hadn't lost staff due to their checkered reputation for being an enjoyable place to work.

The whole capitalist society is based on how much money (some) people have!

Reply on news.ycombinator.com