Reposting because the previous version had a font rendering issue in light mode
251 karma · joined October 9, 2011
Getting good metrics on engineering work is hard. On the one hand, you can build your own pipeline by ingesting API data from Github, JIRA, your CI tool, etc. Then you have to ETL that data, link it to teams and services, and finally select a dashboarding tool to display metrics. That gives you ultimate flexibility at a high cost of maintenance. On the other hand, you can go with out-of-the-box products but a lot of them focus on very specific metrics, such as DORA Metrics, PR throughput, etc. - not necessarily what you'll be interested in tracking, and you might even disagree with the metrics definitions or ethos of the product.
We've gone through several iterations of Okay and we think we now have the right balance between flexibility and ease-of-use. We solve ingestion + ETL and we give dashboarding features, but we leave users free to define metrics using our no-code query builder (with plain SQL coming soon). We specifically avoid forming opinions on what users should actually be building in terms of dashboards - in that sense, we are closer to an observability tool than to "development analytics" products. We are seeing a range of use-cases, such as incident tracking (with JIRA and PagerDuty data), sprint reviews (JIRA / Github), velocity & DORA (mainly Github + deployment data). We are building on BigQuery, DBT for the ETL and our custom ingestion code for API syncs. Our frontend is using Vue with a mix of D3.js and common charting libraries.
We'd love to get the community feedback on this current product approach! This Github app makes it easy to start on our free plan.
In other words, these companies have over-hired as a way to prevent competitors from hiring these same engineers. Although this has created a situation where the company has hired past the theoretical "productive" point, it was still a rational behavior.
Now that the tide is turning, the productivity goal becomes relatively more important than the competitive goal. In the long run though, I don't think the job market will fundamentally change - there is still a shortage of top engineering talent in the US.
In particular, the "KR" part - quantifiable outcomes for the goals - usually helps in clarifying vague projects or ideas that may otherwise harm the team's focus.
It's similar to the famous Warren Buffett's advice on identifying your priorities: pick 20 ideas you have and identify the top 5 goals that you absolutely want to get done; then, throw away the other 15 goals and make sure that you never get tempted to work on them.
The bad reputation of developer productivity metrics comes from the misguided assumption that developers should be measured. The better approach is to treat developers as customers of the management team / engineering enablement team /etc. In that sense, developer productivity is actually a measurement of management effectiveness / organizational health.
Once you build your developer productivity approach on this basis, what needs to be measured becomes much clearer - for example: interruptions caused by meetings, performance of local tooling like local builds, latency of CI, number of on-call pages triggering outside business hours, etc.
The right set of metrics depends on your team and can be sourced from surveys and just talking to people about what's painful on a daily basis. As a quick and dirty solution, I'd even recommend piping Github webhooks and other events to a product analytics tool like Amplitude or Mixpanel. You'd be surprised how fast you can understand things like build or CI latency by using a classic funnel visualization.
A lot of great engineering teams are migrating to this approach, especially when they have a dedicated platform / enablement / productivity engineering team.
Measuring development performance at the individual level is futile given the collaborative nature of the work and its complexity. Team and manager performance are more tractable.
Team performance is highly influenced by external disruptions, which is why the culture of engineering management since the 1980s has been centered around "removing blockers". Focusing on building a smooth-running operation (good hardware and software, fast code reviews, fast deployments, minimum amount of meetings, etc.) is probably what this article means by DecisionOps.
Re: managers, one formula I've used in the past (inspired by Andy Grove I believe) for managers' performance is a combination of the following: how is their team delivering ? How are they helping neighboring teams deliver as well? Do people want to work for this person (internal transfers + hiring) ? Or do people want to leave this team ?
Another example is a manager who noticed that one engineer on their teams was carrying 70% of the PR review load for the team. This was creating a situation where this person was burning out under the review load, but they also assumed that everyone was doing this same amount. We actually see this problem a lot, where people get used to bad situations just out of habit. In this case, the manager re-organized the code review process to make it more fair.
1. Is a good coach / Supports career discussions and discusses performance
"Leader As Coach: Strategies for Coaching & Developing Others" by Mary Dee Hicks and David Peterson. An in-depth, very practical manual on how to coach your reports according to their specific needs and personalities.
"Crucial Conversations: Tools for Talking When Stakes Are High" by Kerry Patterson et al. Helps you prepare and approach difficult conversations with empathy. Especially useful for performance conversations.
2. Creates an inclusive team environment, showing concern for success and well-being
"The Five Dysfunctions of a Team: A Leadership Fable" by Patrick Lencioni. If there is one book to read about team dynamics, this is the one.
"First, Break All the Rules" by Marcus Buckingham. Slightly more general than the 5 dysfunctions, but with a lot of good content about psychological safety and what makes team work.
3. Has a clear vision & strategy / is a good communicator
"High output management" by Andy Grove. Probably the most famous book on this list. It gives you a good introduction to setting goals, communicating about your team's work inwards and outwards. It also features more "system thinking" than the other books - how everything fits together as organizations scale, which is useful for developing vision and strategy.
I'd also recommend reading biographies and memoirs of famous leaders (e.g. Churchill, Marcus Aurelius). Strategic thinking can be easier to learn through examples and pattern recognition than through abstract / self-help books.
4. Is productive and results-oriented / has key technical skills to advise the team
A lot of new managers over-index on the people management side of the job and forget about the key responsibility of a technical leader: choosing a direction, leading others in this direction and delivering results.
"Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations" by Nicole Forsgren et al. One of the most recent, well-researched books into how to measure the productivity and quality of software teams.
"The Art of Scalability: Scalable Web Architecture, Processes, and Organizations for the Modern Enterprise" by Martin Abbott et al. This book uniquely blends the technical, organizational and people management aspects of technical leadership.