Vanity Metrics | Hacker News Reader

Vanity Metrics

firstround.com

79 pointskawera14 comments

Another good technique is to pair metrics. Find a second metric for each metric you care about that you expect to see negatively impacted if you overfit for the first metric. For example, you might pair user growth rates with per user acquisition cost.

The bitcoin world has a real problem with vanity metrics to call out a few specific ones:

It's full of companies like Coinbase and blockchain bragging about number of accounts/wallets but I've yet to see any of them report MAU.

Companies like Bitpay talking about massive use growth using % figures and no absolutes and constantly changing some part of what they report on so it cannot be compared.

SCI/Rebit regularly talks about how they can't give actual figures to their volume because they don't want to attract too much attention yet spend half their time posting to forums with % increases and bragging about how fast they are growing and how huge they are.

Cointip spent years bragging about growth and how much volume they were doing before suddenly shutting up shop.

The only companies in the space that show any real information are the exchanges and Openbazaar and in the case of the later I'm sure they wouldn't if they could hide it.

The important point, though, is that each of these is actually good news for Bitcoin.

I call them input metrics and output metrics - you can't usually directly improve output metrics like 'daily active users' - but they are necessary validation of operational performance (just don't look at them every day)

input metrics are things that you can impact directly (like site speed - which, if improved, will likely indirectly improve the outputs)

from Jeff Bezos 2009 shareholder statement:

360 of the 452 goals will have a direct impact on customer experience.

The word revenue is used eight times and free cash flow is used only four times.

In the 452 goals, the terms net income, gross profit or margin, and operating profit are not used once.

https://www.sec.gov/Archives/edgar/data/1018724/000119312510...

I could easily increase daily active users of even the worst app by paying people in India to use it. This wouldn't exactly help the business - but if that's all you are optimizing for then its not hard to get. In the same vein I could increase people's time in the app by simply making it harder to use. They'd spend more time trying to get what they wanted.

There are no benchmarks that cannot be gamed.

I'm not sure exactly how this relates to the above comment.

But you are making the point that the OP made. You need to pair your metric with an opposing one.

Nobody honest person is going to say "we'll increase time on site by decreasing load time". But on my current project, I've had to take into account that the improved load speed and UX has kept time on site the same. This is actually an improvement as uses are able to interact with the app quicker and therefore their relative active time on site is higher.

Ironically, I have seen cases where increased load time lead to higher conversions by getting people to read the page copy rater than skimming it.

Protip: the quality of any metric used to measure performance will ultimately degrade over time, as the metric itself becomes the target for optimization instead of actual performance. any metric can be gamified

Yes - people start optimizing on the metric, and metrics where correlation doesn't imply causation degrade. The other thing I've seen is competitive pressures degrade metrics. If "A customer using us for X is 95% not likely to churn" is true in 2015, competitive pressure will push it to 85% in 2016, and 75% in 2017.

Also known as Goodhart's Law:

"When a measure becomes a target, it ceases to be a good measure."

https://en.wikipedia.org/wiki/Goodhart%27s_law

Like this blog post.

1) Vanity metric: great. First page on HN.

2) Clarity metric (getting right type of readers and subscribers): not so much. Since I do not think target audience like "sign up" pop ups in the middle of reading the blog.

It was kind enough to give me TWO pop ups - one because I tabbed away to another window for a bit, and then after I closed that a second one when I got 3/4 of the way through the article.

One is at least understandable. The second resulted in me closing the tab because fuck you.

I think it's more that "vanity" metrics are those that get reported by companies to the press, and by higher ups to their employees, because in both cases they want to give the appearance that everything is going great.

In my experience the key decision makers see the reality of the situation and care about measuring what's actually going on.

tl;dr measure the right things, not just those that make you look good.

Reply on news.ycombinator.com