Hacker News Reader

Comment by djcapelis | Hacker News Reader

Noooooooo. All of these metrics are a mistake, don't measure any of them. Measure the things that actually matter about your product and whether or not it's getting better or worse.

Measure outcomes. Only outcomes. Try and make sure your metrics capture the real outcomes and not just what you see in a lab. This usually means you need to use field data to get that information.

View on HN

In larger organizations it can be very difficult for team managers to measure business outcomes of their work directly. Talking to customers is a difficult process, if at all allowed. Information from customers percolates through many layers of support and sales staff, product managers, business stakeholders, project managers etc. By the time the information gets to you it is tainted by bias and politics. Not that it would have helped you much since your objectives are already set by your hierarchy, and again are focused on your ability to execute, not business outcomes.

That may not sound like a very productive environment, but that is the environmental in which most of the world’s software is written. For those engineering managers that have to operate in it, measuring your ability to execute is the best you can do. You might as well do it correctly (which is hard).

And if you’re one of the lucky few working in a less dystopian environment, don’t be too quick to gloat. Enjoy it while it lasts, you’re just one acquisition or reorganization away from joining the rest of us in corporate mediocrity.

I work for one of the world’s largest companies. If your organizational structure is a problem for connecting outcomes with engineering then it sounds like you’ve already found a great problem to solve and so go ahead and tackle that instead of spending time on bad metrics!

(Full disclaimer: I am also currently working as one of those project managers you mentioned. If your project managers aren’t communicating clearly about how engineering impacts outcomes, find new project managers.)

So, based on your public profile, it appears that you work at Apple, which is not at all representative of the experience of the vast majority of software engineers. You also seem pretty young, so you’re unlikely to have a lot of experience prior to your current job. It kind of shows in your reply, too. For example you suggest that a manager should focus their efforts on changing the organization around and above them. That is very bad advice in almost every large company. First, you should focus your energy on helping your team succeed within the organization you have - not the one you wish you had. Then, if you have any bandwidth and willpower left, you can attempt modest amounts of positive change in the organization, starting with the absolute worst hair-on-fire problem you see. But keep your expectations low, because most attempts to change large organizations fail. If you are too unhappy with an organization to tolerate it, and you don’t have the power to fire the people in it, the solution is generally to quit.

You appear to be one of the lucky few not in this situation, but you don’t have enough experience to realize how lucky you are - or how quickly your situation could change. You think everyone is happy with their organization at Apple? Think again. Like I said, you’re just one reorg away.

Please refer to the last paragraph of my previous comment, it is extremely relevant to you.

It isn’t by accident that I am in a functional org. I select where I work carefully, and I walk when they aren’t functional. You should too.

Managing your team is table stakes. Managing 360 isn’t a sign of lack of experience.

First you said that if I don’t like my org, I should change it. Now you’re saying that if I don’t like my org, I should quit. Which is it?

Perhaps you should just stop trying to give me unsolicited career advice, and actually address the comment you’re replying to? Just a thought.

The post looks reasonable to me. The point he's making is that no metric is useful for measuring developer efficacy/productivity & the underlying unstated reason is that dev works tends to be random; problem A is nothing like problem B & there's no way to objectively measure the quality of solution for problem A against another developer's solution.

He's instead proposing metrics that you can use as a high level barometer of project health. What outcome are you looking for? You can't mean the final shipping SW because that's usually way too late. You want to figure out if your velocity is going to let you hit your ship date well in advance of the actual ship date so that you have time to deploy process fixes/hire more people so that you do hit the ship date.

Indeed. Plus, I would add that measuring the outcome (definitely a must-do, not arguing on this) is more about the product. The article was more focusing on metrics for the engineering organization, how it performs as a team. Could you say that your engineering organization is collaborating effectively, only based on the outcomes?

Yes. Divorcing teams from outcomes and their products is an anti-pattern. Focusing on metrics for an engineering organization not linked to the things that organization builds is harmful and will produce the wrong results.

Evaluating teams on outcomes will force them to collaborate to make something great. Evaluating teams on collaboration will just make teams do whatever it is you’re measuring.

> He's instead proposing metrics that you can use as a high level barometer of project health. What outcome are you looking for?

None of the proposed metrics capture project health. Test coverage is the closest and that still isn’t great. (Unit tests are important but you can have 100% test coverage and a 0% working product. In fact, this is how every project begins before the first line of code is written.)

No one gives a damn how many pull requests happen or how many days apart they are.

1) Does the software work yet?

1a) How much of it works?

2) Does it do what we want?

2a) .... everytime? Less? How many 9s?

Report on the things that matter. Track the things that matter. Everything else is actively harmful and distracts your teams from doing what matters.

Also, when you ship the software you haven’t finished, you’ve begun. It is absolutely still useful to keep metrics on outcomes and track improving them over time. It is absolutely not “too late” and any planning that assumes software is done after you ship it is flawed. Features can become done, software never is.

They're useful to measure in that _changes_ to them are indications that something might be happening. Use them as canaries.

For example, your velocity has changed. Why has it changed? Are you bringing new people up to speed? Have you suddenly hit some unexpected complexity? Your mean time to resolve issues has gone down. Are you releasing code with simple bugs you should have caught? Have your team started picking them up sooner?

Whenever something changes there's questions you can ask, the answers to which might well help you do better. Using metrics as an early indicator of change can help.

1) If you need metrics to answer those questions you have a larger problem than metrics can solve.

2) That's an argument for literally any possible metric. You could make an equally passionate argument that looking at and keeping track of free/busy times on calendars or soda consumption or how full the garbage cans are on the floor as an indication and early warning sign that something changed. It doesn't mean those are good things to track. Pay attention to the things that matter. Align whole teams and organizations around those things. Don't let them get distracted with metrics that don't matter.

3) Metrics are a double edged sword and influence behavior. If you're tracking the wrong things, you will get the wrong things.

People would like to have simple proxies for measurements instead of doing actual work. I totally agree all those metrics are useless and only real indicators are "is it working yet" and "are users finding it useful". Because I can make loads of pull requests that are perfect but are not working for end users, and have 80% test coverage for functionalities users never use.

Funny is that as project leader one should click through project as in using it to see if it is working instead of wasting time on finding proxies. If leader/manager does not know how to use system or never clicks through then he is useless...

Reply on news.ycombinator.com