Hacker News Reader

Comment by theonemind | Hacker News Reader

I feel like he's kind of invoking Goodhart's law that when a measure becomes a target, it ceases to be a good measure. Academia can only produce what's possible to produce under the constraints of meeting certain metrics. If you can't structure the work in such a way as to make the right metrics look good while doing it, you can't do the work at all, because you will be fired, and it doesn't matter how important the work is or why you need to do something that doesn't produce the desired metric to get it done.

View on HN

Another way to put this is that the incentives in academia right now encourage "careerism", for lack of a better term. People have to put so much effort into maintaining their position that the actually big picture goals like research become difficult to do, especially when you have to show progress every quarter and funding is only given out in one to two year increments. Researchers end up playing it safe, because big risky ideas may pay off in five to ten years, but that kind of research is hard to fund on a yearly or even quarterly basis.

This. The other problem is that as academia becomes a careerist pressure cooker of competition, where the actual science takes a back burner and you’re incentivized to churn out high-likelihood-to-succeed but not-actually-profound science… why even go into academia?

Personally, I’d have loved to be a part of the Academia of the mid 20th century. But my exposure to research 10 years ago (thankfully, before committing to a PhD) quickly taught me that it’d be extremely unfulfilling, difficult, and poorly compensated compared to private industry. It would be cool to be a tenured professor I guess, but the coolness and prestige of that title is riding the coattails of the past when academia was less of a slog.

I get to work on novel CS problems (distributed systems) in my FAANG job without having to publish X papers per year while I progress on the actual meaty problems, for a lot more pay. It’s just strictly better IMO. I’d rather take 3 years from beginning to end to finish something new I’m actually proud of that’ll be used by hundred of millions of people, than be forced to publish papers very few people will read at a regular cadence or lose my job.

Yes, definitely a case of Goodhearting. One way of further breaking this down is the idea of “slicing the bologna” where if you have an idea, you slice it as thin as you can into as many papers as possible. This you can cite your own previous papers, get more totoal citations & papers, and generally game the impact factor.

Additionally the “publish or perish” regime encourages small incremental research projects, rather than large expensive risky ones that might not produce a paper for many years.

All this against the backdrop that it is getting increasingly harder to find new discoveries since the low-hanging fruit is mostly harvested.

Lol... "slicing the bologna" is such a great phrase to describe this.

I've seen it happen a lot in both experimental and theoretical fields. Rather than publish one big paper that explains a new idea clearly at a high level + show its numerous applications, they publish many (6-10) small results based on the the "machinery" behind this idea. It can be frustrating as a beginner in such fields, like learning from a textbook that shows you just solutions to problems and doesn't explain the general theory.

:) I first heard it in sprinting, where they try to beat the previous world record by the smallest possible margin so they can claim sponsorship bonus from beating it next year too.

I completely reject Goodhart’s “law”. Good metrics are proxies for real things that cannot be gamed. If your metric does not have this quality, delete it.

I like the sentiment, but I'm legitimately scratching my head trying to think of any game-proof metric

Salespeople have fairly ungameable metrics - cash. That is the model we should be shooting for.

Consumer facing web products value depends clearly on number of users.

If a metric cannot be tied to something material, don’t start.

It's never that simple. Salespeople frequently want to do things that damage the brand. A lot of companies force them to present pitches a certain way to try to prevent that.

And you don't want 50 different sales people all trying to sell to the same client, undermining each other's efforts. That means you need some scheme to assign people to opportunities, such as sales territories. They will learn to game that system.

Sales are absolutely gameable and regularly are. Sales on false promises to customers, overpromising feature delivery, overstating financial benefits to the customer, are so common as to be cliche.

The customers end up being short term relationships, the sales provide negative value to the company due to reputation loss (not realized for perhaps years), but the salesperson collects the commission and is probably working somewhere else when the smoke clears.

And I'm not sure if you're claiming "number of users" is ungameable. If so, I'd invite you to reconsider that - we've just lived through an entire era where companies gamed user metrics to harvest sweet, sweet VC dollars. Founders game the metrics for VCs, vice presidents game the metrics for founders, product managers game the metrics for VPs.

If cash is gameable, then play the game. If I hire a machine to generate cash and they are consistently able to do and grow that it seems that this is a good game to play.

You could also reformulate the objective function to NPV.

The point is that not all cash is the same, and some immediate cash causes medium or long term harm. But if cash is your metric, it will be gamed by not distinguishing between healthy and unhealthy cash.

So you modify your metric to try to eliminate the harmful kind. But now you're in the same race as other metrics.

If NPV isn’t sufficient then you are essentially in the realm of maximizing happiness as the objective function.

You're begging the question by assuming we can tell what sales are positive and which are negative value. In other words you're asserting we have perfect knowledge and can't be gamed, to argue the measurement can't be gamed.

Here is a thought, introduce a new term "metric NPV correlation factor". If a particular metric has a perfect correlation with NPV it is assigned 1.0. If increasing or decreasing the metric has no impact it is assigned 0.0. Now that we have established a gradient, it is possible to further the discussion.

You are stating that cash / sales do not have exactly 1.0 but can be something less. I can accept that. However, the correlation is very high and higher correlation factors are harder to game. Metrics with high correlation factors also do not need to be hidden.

Let's contrast this with another metric: number of emails sent per month. Low correlation factor, easy to game, likely to actually cause harm to a business. This is the kind of metric that you would certainly need to invoke Goodhart's law on (i.e. hide) which is, in turn, a sign that a metric should not be used.

My main thesis is metrics must have high NPV correlation factors or should not used at all. High levels of scrutiny should be placed on those introducing such metrics. Finally, no metric should ever be hidden as it is a clear sign that NPV correlation is low by definition.

You've never heard of someone tanking the consumer's opinion of a company in exchange for some short term profit?

If you make the metric "cash over the past 20 years" that would be less able to be gamed, but do you really want to wait 2 decades between each performance review for your sales team?

There's also the problem of one person gaming the cash metric to the detriment of others. Like a used car salesman who lets other salesmen "warm up" the customer but then swoops in to be the one to actually make the sale. Is he really 10x better then the other sales staff or is he jut a leech claiming other people's sales effort as his own?

Even in the best case, where you create a flawless metric that cannot be gamed, it's typically just one aspect of the job.

If people get rated on that metric, they'll maximize it at the expense of the things that can't be measured.

But, there are tons of subtle ways to undermine metrics. Want to be the top salesperson each month? Give negative feedback to candidates for open positions that seem too strong.

Reply on news.ycombinator.com