Hacker News Reader

Comment by karaterobot | Hacker News Reader

The last company I worked at was very metrics-driven, and used a really methodical, numbers-driven approach to make big, unpopular, counter-intuitive bets that failed catastrophically, while ruining morale, and trust in leadership, and gutting the company.

After that experience, my view of metrics-driven product development is that it's a way to offload the burden of thinking too hard or understanding where you're going. When people talk about metrics-driven product development, I think of drivers who blindly follow their GPS' directions into a lake. Obviously, metrics can be a useful supplement to a deep understanding of the product, the domain, the users, and the company, but keep it on a leash.

View on HN

> After that experience, my view of metrics-driven product development is that it's a way to offload the burden of thinking too hard or understanding where you're going. When people talk about metrics-driven product development, I think of drivers who blindly follow their GPS' directions into a lake.

There's also the flip side - metrics-driven development used to keep other people from thinking too hard and scrutinizing work. At my last company, many product decisions were justified using a litany of metrics including AB tests, which people just swallowed because "the data speaks for itself!"

I once took a look at the app we were using to run the AB tests and not a single one had reached statistical significance. The company was basing its decisions on data considered garbage even by the AB app's naive p-value algorithm. I thought that after dozens of A/B tests at least one would be statistically significant just by dumb luck but no, not a single test had even the veneer of credibility.

But what you're describing is noise-driven development. It's not an actual metric value until you've somewhat ruled out noise as the explanation.

I believe that's a distinction without a difference. Most scientists have a hard enough time understanding statistical significance in well controlled experiments and they are (ostensibly) trained for it.

There's so much noise in business data that it's like reading tea leaves, except no one can prove it because by the time they gather enough data they realize that the management and/or market dynamics have introduced even more noise so everyone keeps chugging along, blissfully unaware that they're indulging in statistical fantasies and reinforcing whatever beliefs they already hold.

Accidental or otherwise, we lost something. We had a situation where we had a three legged stool of product/marketing, QA, and Development. If any of the three were too far out of line, the other two could drag them back.

But we killed QA, and Ken Schwaber gave us a treadmill to run on forever where structurally, nobody has time to think past the end of next month. So Product has been running progressively more roughshod over everything for the last 10 years. And since 50% of devs have 5 years or less of experience, that means most people don't know that things used to be better.

I recently had a revelation that merely using a product can often reveal the organizational structure and the cultural incentives that went into making it.

A company that is famous for being metrics driven and having teams that optimise for their own metrics is Facebook. You can tell that the product has "seams". Each page and widget feels like it was individually optimized and then stitched together. Overall it somehow feels off.

Apple puts design on top and has a kind of dictatorial veto over all aspects of the product. Hence products made by Apple feel seamless and cohesive. They are also more polished because when you solely rely on metrics you make it as good as it needs to be and then stop.

Sounds like you've independently discovered Conway's Law, one of the most terrible, true things you can say about software organizations:

https://en.wikipedia.org/wiki/Conway%27s_law

Apple relies on intuition and experience and uses data to support decisions, or not (and data be regularly overruled). Rarely does pure data drive decisions. People coming from other companies usually fail if they can’t adapt to a “data second” mentality.

I feel like Apple’s method is much better than blindly trying to interpret data and let that drive you over the cliff. Data isn’t a replacement for understanding of your product and users.

That's interesting. I was actually just guessing that Apple is not data driven.

I would be grateful if you can share Apples general approach to product development.

The closest publicly available info is an HBR article about Apple’s organizational structure:

https://hbr.org/2020/11/how-apple-is-organized-for-innovatio...

Are you aware of the work by Meehl on actuarial vs. clinical judgment? One of the conclusions of that line of work is that fairly dumb, mechanical data-driven rules ("actuarial judgment") consistently outperform expert opinion ("clinical judgment") even on what we think of as fairly complex topics.

The more surprising conclusion is that the dumb, data-driven rules outperform experts even when the experts get access to the same data and the decision made by the data-driven rule. Why? Humans have a tendency to think they're better than dumb rules and override these dumb rules in ways that make their decision ultimately worse.

I haven’t, but I don’t think it’s a stretch to assume that conclusion doesn’t apply in creative fields.

The funny thing is that people say that about every field, yet one by one, Meehl and the people doing work along the same lines consistently disprove them.

The result has remained remarkably consistent across many fields where people were sure it wouldn't apply. At this point it's stable enough that I think the conclusion is a good base assumption, and it would take serious evidence to show that it does not apply in any given field.

What is a creative field ? Can you write a novel this way ?

Reply on news.ycombinator.com