Comment by EdwardDiego | Hacker News Reader

Comment by EdwardDiego | Hacker News Reader

Parent Full thread

> At one of my employers we had a very large and highly skilled dev community and being quant-biased, we tracked code checkins over time, number of tests written, number of pushes to production etc.

Why are those objectively useful metrics for productivity though? I can think of a dozen reasons why they aren't, as opposed to why they might be. Which, I guess, makes their objective usefulness subjective.

Some people will be working on longer lived feature branches because they're more complex, so they end up with one merge to master in the time it took the dev updating REST APIs to accommodate a new optional parameter to merge four branches.

Someone writes four tests that another dev might cover in one because the four assertions were all closely related to the same invariants and code paths. Who was actually more productive?

One developer pushes to production three times in one day because they typoed a property name twice... ...etc. etc. ad nauseum.

If you seriously believe that these metrics are objective methods of measuring developer productivity, I strongly suggest you spin off a business, because you're about to make millions.

It seems to me that while any one of those metrics isn’t particularly useful in isolation, if the same person is consistently at the top of all of them, that’s telling you something.

_What_ it tells you, however, depends on whether or not the combined metrics are used to evaluate people (or more accurately, whether people _think_ they're being evaluated by those metrics). Goodhart's Law[1], ruining everything since 1975.

[1] https://www.lesswrong.com/posts/YtvZxRpZjcFNwJecS/the-import...

Maybe. but what if their peer has far fewer commits to their name, far fewer written unit tests attributable to them, because they spend most of their time using their indepth domain knowledge to assist other developers, who then make commits based on that contribution?

Do they have to sit around and ensure that their coworker includes a "Co-Authored-By" in the commit message? One commit message? All commit messsages? etc. etc.

This guy was also top in terms of the number of FAQ entries written to help people. In fact he wrote the FAQ system itself and also wrote a right-click plugin so if you rightclicked on a question in chat it would auto-search the FAQ.

See now that might be a better metric.

Number of check-ins, if the check-ins are high quality as he implied, are a pretty good rule of thumb to measure the productivity. The guy working on the most complex feature with only one merge to master should check-in even more often than the guys working on the simple features.

I don't understand that logic - the complex piece might require tons of research, experimenting with different approaches, benchmarking etc, before getting to something committable. And the successful result might only be a small number of lines.

I worked on project where I had dozens of commits every day. Now I work on project where I have one commit per few days. Sometimes I need to research for a week just to create one small commit.

And I am still the same person - only my position changed.

What is the point of checking in a horrendously incomplete feature that nobody else is going to work on, that probably cannot be tested by anyone until it's actually mostly done?

I'm working on such a thing, and I will check in ~100k lines of code when it is the right time to do so. That is not now.

In a CI/CD setup, you could make lots of atomic commits growing the feature behind a flag.

A side effect of atomic commits is that number of commits does better correlate to "work done", I suppose.

You could, but does that really solve any real problem? Now you're wasting more time on CI, commit messages, and trying to keep them atomic instead of delivering the feature. It's not even clear what makes up an atomic commit. If the feature does not work, it's arguably not very atomic. Stubs, mocks of parts that need to be mocked to push forward now and redone later properly, unimplemented branches that end up in abort(), these are arguably not very useful history either.

Also, I disagree that number of commits is a useful measure of work done. Not any more useful than butt in seat time. Nobody can say whether that feature is 200 or 2000 commits, it really depends on how you split it up. It could be 2 commits, plus the fixes that follow once it actually runs and gets tested and fleshed out.

If tracking progress is a priority, you break the job down to some reasonably small number (no more than a couple dozen) of items in a TODO list and update it with what's done and rough estimates of what's left to do. That's a hundred times more useful than "N commits, 123 hours of butt in seat."

Or maybe you can just demonstrate what works. Or tell, when it comes up in a meeting.

(I'd argue that butt in seat time is actually more useful than number of commits, because the latter can change by an order of magnitude depending on how you split it up but I've never estimated to take a month to deliver something that ends up taking 10 months.)

This is probably not the place to debate the pros and cons of CI/CD in general, but if a team is already doing it, isn't the list of atomic commits effectively the TODO list you're talking about?

Of course, over-engineered solutions will score you more commits (or more lines of code, or more TODO items, etc) but still be a bad outcome.

If you don't know how many commits it will be before you start, no, it is not a TODO list. I've never worked with such perfect knowledge. Usually, there's a high level understanding of what it takes to deliver a feature, which allows for the creation of a coarse TODO list that works for tracking progress & coordination. And then there's the devil, in the details. One commit is not one feature, and there are too many details for an a priori list of all the commits you're going to make. And very often you end up doing a bit of opportunistic refactoring & cleanups & little fixes to related code when fitting the new stuff in.

Even very small and simple features can end up being anywhere from 1 to 15 commits.

Yes, you're right; it's not a TODO list because it's not known ahead of time. Therefore, it can't be used to track progress on the particular project itself.

This all came up in the context of measuring productivity, though, not measuring progress on a specific project. Looking backward, it is a list of items completed. If the teams strives to keep them atomic in general, they are more likely (though by no means guaranteed) to be closer in size. So my guess is that this would correlate better to productivity than lines-of-code or time-in-seat, while still falling well short of the ideal.

By check-in, do you and the OP mean commit? I had interpreted it as a feature branch being merged to master - in the days of Git über alles, you can commit like crazy.

No developer metrics are great, especially not if devs are incentivized to game them. However, I wouldn't be surprised at all if there's a correlation between good developers and those metrics.

If you incentivize the metrics, I would be surprised if the correlation held.

Well I suppose the correlation would still hold for the good developers but it would then also hold for the not as good developers. Over time maybe the bad developers would show a higher correlation than the good developers, but maybe not.

Reply on news.ycombinator.com