Hacker News Reader

Comment by schoen | Hacker News Reader

Your observation is related to something called Goodhart's Law: https://en.wikipedia.org/wiki/Goodhart%27s_law

I guess thinking about this increases my sympathy for search engines and spam filters not being transparent about the algorithmic bases for their rankings, even in a week when I've had some personal e-mail blocked as a spam false positive. The resources that are getting put into discovering how to game those scores are staggering.

I assume there are also exceptions to Goodhart's Law in terms of metrics that effectively can't be gamed, or that otherwise better manage to align different groups' incentives.

View on HN

Also https://en.wikipedia.org/wiki/Perverse_incentive .

The best case in business is when you tell a sales professional that they will get a percentage of whatever business they bring in, judged the moment when it's in accounts receivable. It's difficult to game a wire transfer from another company.

Education is very different than sales. I don't know what the solution is, especially when measuring "learning" itself is very difficult if not impossible.

> The best case in business is when you tell a sales professional that they will get a percentage of whatever business they bring in, judged the moment when it's in accounts receivable. It's difficult to game a wire transfer from another company.

I've seen the result of this. The salesperson oversells, and the engineering team death marches to attempt to build what the salesperson has already received their commission for.

Incentives are tricky.

"especially when measuring "learning" itself is very difficult if not impossible."

That's ridiculous. Math, science, vocabulary, history, and reading comprehension can all be tested effectively, with little risk of "gaming" it.

(at least at the level of a tpyical high scholl curriculum)

Especially high school math can be gamed effectively. For "word problems" you can drill students to look for keywords and they will arrive at the right result without understanding the problem.

My wife teaches at a 4-year state school and tells me that one of her students did just that, she looked for plausible keywords but then didn't know what to do next, this was chemistry, not math.

Reading comprehension is harder to game.

> you can drill students to look for keywords and they will arrive at the right result without understanding the problem

> she looked for plausible keywords but then didn't know what to do next

That doesn't sound like she arrived at the right result.

Test-taking strategies are definitely possible without learning the material. Hunting for keywords is a common one; learning to recognize the kind of answers that show up as confounders is also pretty common. Even free response questions can be gamed, if you learn how to come up with plausible answers that can rack up partial credit.

It is difficult to game your way to an A without knowing a fair amount of the material, but it's definitely possible to get a passing grade and possibly as high as a B. And students are often very eager to try to claw as much partial credit on a test as possible.

You might think that. But take it from a former teacher, it's actually very difficult assess learning. It's an entire field of study in itself.

[1] https://en.wikipedia.org/wiki/Educational_assessment

This comment is a fantastic example of not knowing enough to know how ignorant you are - to start with, you appear to be conflating “knowledge” with “learning”.

It's not that hard to think about how to game that and the culture and incentives that will develop outside of very limited utopian circumstances.

The logical outcome is that the biggest parasites will look to attach themselves to the biggest flows of money with low levels of elasticity relative to their input.

And they will do whatever they can through side-channel methods and attacks, whether it be destructive to other employees or the company itself, to stop others getting into the position, including navigating/manipulating the hierarchy to maximize flows through them.

Half of finance/executive remuneration is literally built around that strategy (manage big flows of money unrelated to your personal input), and I'd guess a huge number of toxic companies and employees are directly attributable to people who believe they've successfully implemented that incentive scheme.

Except we all know what happens in that case. Sales promises whatever they need to in order to get the sale, and then engineering is left to pick up the slack, working unpaid overtime to compensate.

Reply on news.ycombinator.com