Useful engineering metrics and why velocity is not one of them
lucasfcosta.com
lucasfcosta.com
This is a qualitative process, though, and using velocity metrics from your project management tools is like looking for your lost keys under the streetlamp because that's where the light is brightest.
It is also important to recognize the importance of keeping slack in a system and not over-optimizing it, as explained in Tom DeMarco's excellent book Slack: Getting Past Burnout, Busywork, and the Myth of Total Efficiency.
^ 100% agree.
Tracking output metrics (including LOC, # of PRs, velocity points) doesn't help you identify or fix poor tooling, culture, or processes.
Metrics like cycle time get a little bit closer, but still fail to account for team context or surface root causes.
Particularly when a team has multiple projects, looking at each project's velocity helps you see if you're smoothly building out new features, getting bogged down, distracted by bugs in an old project, consistently underestimating the effort required for all things, etc.
It's a terrible target, not good for evaluation, but a handy metric for a PM to quietly check and use to make sure things are proceeding smoothly. The PM should probably know by other means but no one is perfect.
So what? The question isn't "what helps you fix everything?"
The question is "how useful is velocity?"...without qualification as how you quantified the values. Looking deeper, the question naturally progresses to "what is the most useful metric?" or even "what is a useful metric?" in determining performance (a scary thought experiment for most developers).
Narrowing the scope of the analysis is a necessary step in determining value. Let's dispense with the gaming numbers strawmen and refactoring and testing rigor etc etc.
Theoretically, if it's you (alone) on a project and you have no other external signals about how well features are valued, what singular value, over time, would you look at and judge your progress historically? This is the closest you can get to an abstract vision of (personal) performance on a project divested from all other concerns. I can always look at my LoC.
That's not just a poor analogy but wrong. I can't even start to identify what aspect of velocity is the key and the street lamp or searching. Velocity is a tool to assess the amount of work you do per cycle and to help plan how much work to take in a given cycle.
It has nothing to do with losing key,, searching or only looking where there is light.
As usual, for any process, haters will just trade jabs and nod appreciatively and how much it all suck. Assessing how much work you team does every cycle is a useful metric, and people higher up like to have at least a small glimpse of how much work and and much time it will take to finish a feature.
One useless metric though, is snark.
I am curious if you also track UNPLANNED WORK.
The reason I ask is because in my experience a VELOCITY metric is in fact useless without also tracking PLANNED vs. UNPLANNED work. I am hoping you say "Yes, we track that".
BTW I thought the metaphor was quite appropriate (and hardly snarky -- not sure why you read that into it).
The problem with velocity isn’t poor metrics, the problem with velocity is it’s trying to measure a meaningless quantity.
I think the word choice of "velocity" indicates that its original intention was to have direction, otherwise it would be "speed" :).
What you're defining as velocity here is the exact 'micromanagement software tracking velocity metrics' that the parent comment is arguing against.
Velocity is probably more accurately thought of as the rate at which we can improve or create the right products for our business goals. But as the parent said, this is a qualitative and difficult to capture with metrics.
But to help clarify the point, if the project knew what was useless then people wouldn’t be working on it. Essentially, the team is defining useful work by what’s being worked on which robs the direction of any meaning.
When I hear such things I have the strong urge to factor in how lasting such solutions are.
A company that puts up 100 bridges a year of which 90 collapse within two years is a worse engineering company than one that puts up one a year that lasts for 50 years.
It's a similar property to "code quality" in that regard. There are metrics like 'test coverage' or 'dependency graph complexity' but these can also be gamed intentionally or unintentionally much like LoC.
There are lots of descriptive metrics you can come up with that can be useful in exploring how you are performing or how your performance is changing with regard to quality or velocity. But treating such a metric as an actual proxy for the underlying property is not useful and sometimes counterproductive as you point out.
I think this is where the notion of software craftsmanship appears. We are still at the point where it requires an expert who is engaged in the process to essentially sense how they or their team is performing on these properties. They can use metrics to inform that sense or to help in determining what actions should be taken.
But this is frustrating for larger organizations where frequently the people who want to measure a certain property are not actually participating in the process because they are separated by multiple layers of management. You essentially have to trust that every management layer below you is able to accurately assess a qualitative property, which is difficult to standardize and scale.
that's exactly right, it's taking 2 dimensions (estimate and fitness of estimate) and quashing it down into 1 dimension, where you necessarily lose information.
It's a point in that it's an abstract idea that can represent an atom or a universe.
It's not useful for anything outside of the abstract.
Velocity is important but I think it's an orthogonal concept to quantifying useful output. In my mind, velocity is how quickly developers are able to unblock themselves and overcome obstacles. It's very possible that people can have extremely high velocity working on pointless stuff but that's not what velocity is meant to track anyway.
Tasks vary drastically from project to project,including due to varying requirements, ambient software environment, technological changes and a host of other factors. Metrics are a way for project managers to pass the buck on to the engineering teams and nothing more.
Unfortunately what everyone forgets is that historical information only applies if you are doing "a thing" that is similar to what you've done before.
A: what is your budget? What other work will you sacrifice?
How does a longer period of time help with estimation?
One way to address this problem is to not ask the question, but
to work with your teams to set goals for a longer period of time ...
Then structure your organization such that it doesn't depend on
the exact timing of work completion.
I actually mostly agree with the comment I was replying to, I just think the “set goals for a longer period of time” was not filling in the whole picture. You can’t just set the same types of goals you had when you were making sprint estimations, stretch them out to be quarter estimations, and expect the problem to go away.
Given that so much of what we do as developers typically depends on more than just ourself in an effort, I agree with what I think tadfisher is expressing; realistic delivery planning can be achieved by focusing on "teams and goals for the next few months" and cannot be by trying to answer "how long will <insert feature never done before> take?"
There's a fog of uncertainty that gets impenetrable past a certain time horizon. We need businesses that are capable of forging ahead into the unknown, being honest about the level of confidence given our knowledge at the time. I don't see that approach coming out of MBA schools, and I think Elon's approach is in marked contrast to that.
In many ways old school grassroots agile has penetrated the lower levels of the org (to some degree), but the upper levels haven't changed in decades, and this impedance mismatch is most visible in the rigidity of planning (which is also forced on them by outside pressures).
Code (implementation) is an integral alert part of the software design and planning process, and you can’t know how long something will take until that’s complete.
While estimation seems to be a shibboleth of certain types of project management, the fact that implementation time estimates are rarely accurate - and often significantly wrong - suggests that they are not as vital as we’re lead to believe.
I disagree with your conclusion. Just because we're terrible at estimating doesn't mean it's not important.
"100% uptime is rarely achieved, which means it's unnecessary and unimportant".
> Code (implementation) is an integral alert part of the software design and planning process, and you can’t know how long something will take until that’s complete.
McConnel talks about it in "Software Estimation". The only way to accurately predict anything is to base it on your previous experience, which I allude to. Unfortunately (or fortunately, depending on how you look at it), even projects that looks similar on the surface are more frequently dissimilar than similar.
Regardless it's useful to be able to answer the "how long" question at least a coarse level to see if it's worth doing something.
Recently posted on HN was a link to a post at Lunar Logic where estimation is basically broken into 3 categories: “NFC” (no clue), “TFB” (too big) or “1”. They explain why here [1].
This has been my experience over more than 30 years of building products. The duration of any given feature/story is effectively unknowable, but we can make a decent guess as to it’s achievability, we can get a decent idea of the rate at which stories are completed.
The main problem I have with more detailed estimation is that the estimation process itself consumes significant engineering resources which IMO should be assigned to actual engineering.
To your specific points:
> "100% uptime is rarely achieved, which means it's unnecessary and unimportant".
This is specious. As an industry our estimates tend to be wildly wrong, and despite decades of dealing with “the software crisis” they are not improving. My position is that this means that we’re spending a lot of time estimating and then more time dealing with them being wrong. It’s just a waste.
> Regardless it's useful to be able to answer the "how long" question at least a coarse level to see if it's worth doing something.
This seems to be the wrong way to do it. Surely the approach should be to ask what value this “something” has to the business, and then determine if it’s technically feasible within that value.
But that would mean more work for the executives, and quite frankly this, in my direct experience, is not the kind of responsibility they like to take on.
[1] https://blog.lunarlogic.io/2016/how-we-estimate-at-lunar-log...
The metrics “work,” in that they’ll go up, but the things you don’t measure will get worse, often (eventually) catastrophically.
In the case of velocity, you’ll get people taking shortcuts and sacrificing quality, both internal and external, so they can juice their numbers. The outcome is technical debt and arguments about what it means for something to be done, resulting in slower progress overall.
Source: I’ve been consulting in this space for a few decades now, and have seen the consequences of using velocity as a performance measure time and time again.
(Other metrics are just as bad. See Robert Austin, Measuring and Managing Performance in Organizations, for an explanation of why knowledge work like software development is incompletely measurable and thus subject to measurement dysfunction.)
Metrics are great to diagnose the overall process and get a sense of what's going on that can be superior to our qualitative feels about it.
And metrics can also spot an occasional outlier or performance problem. Used sparingly, this does not encourage juicing the numbers.
Some people don’t know how to manage without measurements, punishments, and rewards. It’s a correctable flaw. Measurement-based management is called “theory X” management, but knowledge work needs “theory Y” management. There’s a lot of material out there on how to do it, including a section on it in my book.
The only way software development metrics are useful is to get an indication of the performance over time of the same team working on the same codebase. That should give some indication of the overall trends, but when the numbers start going down, how will you accurately determine the cause of that? Will you treat the developer team as a black box and insert more probes, or will you talk to them and rely on their qualitative assessments after all?
Anyone who's built software knows that estimates are guesses and are bound to change once the work actually begins (not to mention if scope changes midway through).
Additionally, velocity does not "count" work happening outside of tickets, e.g., helping with a customer support issue, assisting another developer with legacy code, reviewing other people's work, spending time on feature planning.
Your capacity is adjusted accordingly. In my team an engineer who's going to work-on production support, would not have his capacity accounted. Similarly, if an engineer has a lot of cross-team work going-on then we'll reduce his capacity as well, and so on.
I believe I was not clear: it's a rotational support - where every engineer is encourage to spend a sprint worth of time in every quarter on working on support tickets. You've complete autonomy - you don't want to work on support tickets, it's alright. No one is forcing you. You want to spend next two weeks on a training or self-learning - go for it.
> > Your capacity is adjusted accordingly. ...
> That level of micromanagement is demoralizing and it hurts the company.
Encouraging engineers to "spend a sprint worth of time in every quarter" or the "next two weeks on a training or self-learning" is laudable, but does not qualify as providing autonomy or the lack of micromanagement. The reasons why are A) the decision is still solely yours and B) clearly time-driven instead of collaboratively prioritized.
You've complete autonomy ...
> This level of "accounting" does not convey trust in an engineer, thus demoralizing them by way of eliminating their autonomy (freedom from external control).
I'm curious to know how it's done at your workplace. Please share.
I’ll bet $5 that people conform to your model by working longer than necessary and your “velocity” is considerably lower than it could be, despite being measurably consistent.
Similarly, managing a product team is also managing a running system; that system is just made up of meat sacks whacking plastic buttons with their bony protrusions, rather than computers humming away in data centers. A product manager still needs to quantify that the system making that product is working correctly, and metrics are essential. Otherwise you will only be guessing as to how the system is working, and those guesses will be much more haphazard as the system grows.
But this does not look like engineering; looks more like statistic process control applied to software dev
Not engineering!
I’ll consider adapting the title, but not sure whether folks would grasp my intent as quickly
Velocity has a strong definition in engineering that has been borrowed and misused in software management.
I expected the article to be about accelerometers and how the ill effects of vibration have been mitigated
as a software engineer, I really apprecaite this.
> about accelerometers and how the ill effects of vibration have been mitigated
I really hope I can mitigate vibrations of mood, esp the procrastinatory kinds. But then I don't want to over-"engineer" my behaviours or it would start to become existential. Like sometimes I don't even know what is meaningful anymore (despite working on my dream job), or if I can truly humanly love myself. Being human is tough.
Hope you're doing okay, dude. Meaning doesn't mean much, but learning to love and be happy is (from what I can gather) far more important than the zeroes and ones we tinker with. I'm trying to figure it out myself and waiting for the good times to come around again, too.
Thanks. This hits me hard. I think I've been focusing too much on "meaning" (and trying to do the meaningful things, etc) and over-exhausted myself. I think I need to practise loving and being happy more. That turns out doesn't come automatically.
Of course, it's not engineering; it's all about metrics, which are statistical
For better or for worse, though, most of the modern software industry has converged on "software engineering" as the name for what they do, while terms like "web developer" usually invoke spooky prehistoric (20th-century) connotations of companies that treat software as "part of the IT department" and a "cost center."
Now everyone is a tech/dev advocate because they have 500+ linkedin connections and repost the same cloud news everyone has already read.
I'm just going to coin the next stupid title now, 'cloud astronaut', they're so good they're beyond the cloud. That will fill the blank void that is your ego. You're welcome.
This statement FTA is completely antithetical to agile development:
"A manager who wishes to make their team’s cycle times more uniform can try matching the rate at which tasks enter the system to the rate at which they leave."
On an agile team, "the manager" doesn't moderate the work rate; rather, the team measures their own work rate (approximately) via velocity and manages themselves accordingly.
You could certainly think of the sprint board as a kanban board with a finite lifespan, but I don't think there's any benefit to doing so.
By keeping the backlog prioritized at all times, and pulling from the backlog instead of a 'sprint backlog' - it avoids priority inversions. Planning what will be your highest priority, that possibly changes mid-sprint, is a priority inversion.
Kanban works well in all situations where sprints work well. Instead of trying to predict in a planning, you just do the work instead. Any predictions are still the same based on looking at historical data and comparing to the backlog, just less waste involved and less subterfuge to try and get developers to work overtime (I'm not aware of any sprint planning that really takes into account error bars, so the prediction/planning is BS & management planning porn, nothing more)
The purpose of sprints is not to divide up the work---this is a common misunderstanding. The purpose of the sprint is to set incremental goals based on incremental learning. So instead of setting a goal to create the complete software system from the beginning, you set a goal that can be accomplished within a fixed timeframe that people can make reasonable estimations in and that will test some of your key assumptions about the software.
Simply throwing everything into a massive backlog and grinding through it indefinitely is a recipe for failure, since there is no opportunity to reflect on the larger picture: the team is too busy focusing on the assembly line and not paying attention to the product. And of course you can still have "priority inversions" in kanban, but unlike the finite sprint, you may go months before you realize what you've done wrong.
As I said, there are appropriate times to do kanban, but the situation you describe is not one of them.
> Simply throwing everything into a massive backlog and grinding through it indefinitely is a recipe for failure
Why is that? Scrum has the same backlog.
> since there is no opportunity to reflect on the larger picture
Why is this the case? Kanban often says that you talk about retrospective items any time they come up. You then even can re-prioritize such items right when needed, meaning you solve todays problem today, not tomorrow. Retrospectives in scrum can often come too late (and personally I've never really seen them work well, too little, too late, and often compete against 'business' priorities that always win out).
> And of course you can still have "priority inversions" in kanban, but unlike the finite sprint, you may go months before you realize what you've done wrong.
Kanban is less susceptible to priority inversion. You keep the highest priority at top, it is evaluated more often, and you pull from the top. Why does this mean that months can go by? Why is that any more the case than say scrum where you are encouraged to fix priority for weeks at a time?
> As I said, there are appropriate times to do kanban, but the situation you describe is not one of them.
I think the burden of proof is on you to state places where scrum uniquely bests kanban. The statement that kanban is best for places where ad-hoc work comes in constantly is a case where scrum spectacularly fails. For regular project work, kanban can be a great fit (I'd say universally better than scrum, since planning is just complete BS. Kanban is often little more than scrum without the planning sessions and with far more frequent prioritization, and you get to keep all of the same retrospective & estimation tools)
But I'm confused by this statement:
>planning is just complete BS
To me, planning is where design meets reality. Planning involves several activities, but probably the most important is prioritization. I'm curious why you think that prioritization happens more frequently in kanban, where the convention is to simply take the next task off the stack. Are you saying that every time someone takes a task off the kanban backlog, they independently re-prioritize the entire stack? Or do you have regular re-prioritization sessions with the team? If it's the former, that seems like a lot of overhead and opportunity for conflict and misunderstanding. If it's the latter, how is that really different from regular sprint planning? Because surely, as part of the prioritization, you would also consider things like estimation of difficulty and risk.
> So instead of setting a goal to create the complete software system from the beginning, you set a goal that can be accomplished within a fixed timeframe that people can make reasonable estimations in and that will test some of your key assumptions about the software.
In Kanban you can do all of that, except faster. Instead of waiting for the arbitrary sprint boundary, you can immediately evaluate an item that has landed in production and immediately re-prioritize the backlog based on any learnings. That would be compared to: we will deploy this in sprint A, then in B we will see the results, and then in sprint C we will work on our updated items. The latter is a many week delay compared to assigning tasks in strict sequence of "ship/measure/adjust".
In the worst case, you ship at the beginning of a sprint and then instead of working on the next priority thing (that you learn from having shipped), you instead keep working on the other sprint items. At that point, it could be waterfall, working on items that you now know are pointless, but you still have to anyway because it was planned some time prior to do so.
Kanban doesn't forbid regular retrospectives. You can still pause frequently, reflect, and update your process accordingly.
The purpose of sprints is to try to estimate the completion of a release -- and as progress is made the drift from that original prediction. Sprints in theory let a manager estimate a team velocity. Together with estimates for the task in a release, that, in theory, gives one the ability to predict when a set of features (a release) will be complete. I have yet to see this actually work, but it could just be reflective of where I have worked.
This is not the purpose of sprints. The sprint is a fixed increment of time intended to encourage incremental development of complete software systems. The idea is that while people don't have an intuitive idea about how much work will have to go into developing a complex application, they do have an intuitive feel for how much work they can accomplish in a couple of weeks. So instead of asking, how much time will it take us to implement a fixed set of features, the team will be asking how many features can we implement this sprint?
>Sprints in theory let a manager estimate a team velocity.
A manager is not a consumer or producer of team velocity. The team itself is the only legitimate consumer and producer of team velocity. Agile is about team self-management.
I've always seen that mainly accomplished by breaking large stories into smaller ones. What do sprints add to this?
Even in a reductionist approach, though, sprints provide a regular opportunity to review the appropriateness of the breakdown or the validity of the parent story.
In my experience, the fixed time increments have always felt too constraining.
I think a team can accomplish a similar goal of having regular incremental progress in the form of complete systems by relaxing the fixed-length sprints requirement.
The alternative that I favor is Kanban with defined releases. A release, as a collection of features, is a complete system, but it doesn't necessarily have to be finished in the same time that the previous release was. A larger release may take six weeks instead of the smaller previous release that only took one.
You make a lot of good points, thank you for sharing your thoughts.
So, to paraphrase the distinction, instead of death marching for a large set of features without any feedback, the team decides "we won't do more than 2 weeks of work at a time before we ship & deploy some feature."
(I really wish I could find the link) A recent hacker news article looked at this situation and noted how this does not really allow for extensive R&D where there is no deliverable after the end of two weeks. An example was Google BigTable which had no deliverable at all for 8 months. Yet, the investment into BigTable lead to a very huge pay-off, and the point made was that there was no good way to break up the R&D required into two week deliverables, and the ROI of that work was far more than most any 2 week increment.
Overall the issue for scrum is context, it has its places where it works well - but it is not the one-size fit all solution that so many places try to make it out to be. For example, two weeks does not always allow for a lot of rework, experimentation, architecture, and R&D. A team can choose to make a deliverable be "the results of R&D", is this still fungible thinking and is not actually delivering a release.
Personally I would be a lot happier whenever a scrum master and/or product owner/manager that run a scrum team could clarify exactly when scrum is a bad fit (this feels like the expert-beginner problem). If a person knows when scrum is a bad fit, they can at least try to mitigate those problems and/or lean on other development methodologies. It is almost like silo'ing into one programming language and using only that language despite it not being the best tool in some situations (and also of detriment, various alternative techniques are not learned; eg, sometimes a bash script is the right tool to use, and not branching out and trying to solve some problems with a full blown programming language leads to a lot slower of a solution. To illustrate this, Amazon once had a team spend time to find all phone numbers on its website. This took a long time and the program was complex and never worked well, a very competent shell scripter got wind of this and solved it in 15 minutes with a bash one-liner. This became an interview question at Amazon at one point even, in part to check if a candidate will realize that their language of choice might be a bad fit for the problem at hand)
Sure, you can. But it requires discipline to actually do it. It's very easy to get wrapped up in a task that runs over, and when you multiply that tendency by everyone on the team, the idea that it will actually get done regularly in practice is optimistic, to say the least. And if you are doing it regularly, together as a team, is that really so different from a sprint?
Conversely, the notion that you can't change course mid-sprint is simply wrong---you absolutely can. The concept of sprint commitment is an acknowledgment that there is value in completing something workable, even if it's the wrong thing, as opposed to staying constantly in a state of WIP. But if it's obvious you're doing something no longer needed or technically feasible, nothing in scrum says you have to complete the sprint as planned.
You can't really do this without having some idea of how long each item in the backlog will take. If feature A is twice as valuable is feature B, should A or B come first in the backlog? Depends if A takes the same time as B, or four times as long.
(ignoring various other considerations about how you prioritise, and just focusing on delivering value here)
This is why rough estimation is part of the planning process in XP. The developers estimate stories so that the product owner can prioritise them. The estimates don't need to be precise, just roughly right relative to each other, to act as a denominator.
Since the developers are only pulling from the top of the backlog, the product owner doesn't need to prioritize the whole thing, just enough to keep the team busy. That means the developers only need to estimate stories which the product owner is considering for prioritization.
You could try to do this continuously - a developer needs a story to work on, so they ask the product owner, who picks a few candidates, then asks the team for an estimate on them, then selects one to be next. But that would involve everyone interrupting everyone else quite often. So typically, you get together once a week and do this until you have enough stories to keep the developers busy until the next get-together.
You can tell, roughly, if you have enough stories for a week by counting how many size units of stories the developers get through in the previous week - that is, your velocity.
Nothing in this way of defining priority is really any different between scrum & kanban. Perhaps the only key difference is when work is allocated to be done, scrum often focuses on 'how long will it take' to avoid "assigning too much to one sprint". Hence, a low ROI item could certainly be fit into a sprint simply because of time constraints, even though it might be really low on the overall ROI list (thus a priority inversion where the wrong thing is worked on simply because it fits an arbitrary time period).
> You could try to do this continuously - a developer needs a story to work on, so they ask the product owner, who picks a few candidates, then asks the team for an estimate on them, then selects one to be next.
If you have your backlog prioritized (or at least deep enough you can pull work for a while), then doing this continously is cake. The developer pulls from the top of the backlog. When a new task or a story is introduced, it is stack ranked. When and how often you introduce new stories in kanban is really up to you. So it need not be a constantly disruptive process. Typically new stories are looked at on a daily basis, which allows for a lot of responsiveness. At the same time, if knowledge changes about a story, it's priority rank is changed right then. It does not take very long to do stack ranking.
> You can tell, roughly, if you have enough stories for a week by counting how many size units of stories the developers get through in the previous week - that is, your velocity.
Tl;dr, velocity makes some interesting assumptions and uses bad math
Velocity does not take into account error bars. This is bad math! Each story has an error bar of give or take 30% (assuming it is well estimated with Fibonnacci). EG: An '8' could be a 6.5, or could be a 10.5. Yet, somehow by adding up the expected size of stories together the overall accuracy increases.. This is an oddity and bad math. Velocity should always be reported with a plus or minus, yet, it doesn't.. Somehow all of math behind propogation of error is assumed to cancel out. Beyond bad math, velocity only says how well you did last sprint, and "past performance does not guarantee future results." In short, velocity assumes that people can estimate. The only tasks that will have reliable estimates are ones that are very similar to something you have already done many times (eg: how long would it take you to brush your teeth with this new electric toothbrush? You know how long it takes to brush teeth with a convention toothbrush, so your estimate would be pretty good). But let's say, how long will it take you to learn Electromagnetism? Cool, you just learned that, how long will it take you to learn Calculus III? If you've taken Calc I and II, you would perhaps have something of an idea, but you're still to be really way off on estimating that.
You could do this, but it would be extremely silly.
> This is bad math!
Since the only purpose of velocity is to know roughly how many stories to prioritize in the backlog, none of this matters in the slightest.
Given error bars are not taken into account, the proposition is that velocity is incredibly inaccurate. Presumably, most of a backlog is prioritized (otherwise a high priority story could be at the bottom, leading to priority inversion). I thought the purpose of velocity was to know commitment size for a given sprint (how many stories to bring into a sprint). Given that the inaccuracy of velocity is not taken into account, nor the compounding of the uncertainty, knowing how much to commit based on an inaccurate value leads to a lot of "failed sprints".
Re: prioritization, how you evaluate priority is completely up to a team. Anecdotally, Maesk ran agile and did not take into "cost of not completing a story". By that measure, there were three stories that were worth over 100x all other stories combined. Yet, they delayed deployment and efforts of these stories in order to "fit" in other stories that had low effort. Logically, with that large of a benefit, they should have stopped literally everything else and gone all in on those 3 stories. If you're really interested, I can try to dig up the youtube talk that went into this anecdote. The speaker in that video also noted that the data they looked at suggested the benefit of stories usually was magnitudes more than the level of effort. Hence, at times it almost does not matter the level of effort since the return is a better proxy for overall value.
Prioritisation can be whatever you want, scrum does not say what 'priority' means. That can be as silly, or as complex as you want. There are many inputs to priority. In practice there seems to be an over-emphasis (near-obsession) over 'cost' rather than value provided. A good example of this are items that have deadlines to them. At some point, the "cost" of implementation is not the biggest factor, yet it often gets the most attention and time allocated (vs say, dollar cost of not implementing per day)
My point with velocity is that it is inaccurate and using it will be just wild guessing. It's like saying, it's 65 degrees out - "cool, I'll wear a t-shirt". But, it's 65 degrees out plus or minus 30 degrees. As an engineer/scientist, we should already know that any number should always have error bars reported, otherwise it is meaningless. Similar to the temperature, you can look outside and infer the temperature through other means. It is likely then that "velocity" is noise compared to the true means we use to know if we have enough in the near backlog or not. Worse yet, velocity is often used as a cudgel to force overtime and declare a sprint "failed," even though velocity is so uncertain (yet is not treated as an uncertain value).
Kanban is an alternative to sprints, ...
... for contexts in which the team is persistent across projects
and it makes sense to let them manage their work apart from the
projects that they contribute
So sure, if you:
- release usable software
- collect feedback from stakeholders based on the release, and
- re-design and re-prioritize the perceived work remaining
on a regular basis as a team, then you can use Kanban for a single project. But it will look an awful lot like sprints, because it's hard to do all that on a continuous basis, it would be a lot of overhead and context-switching to do it daily. On a project of any size, you'd have to do it all periodically. Say, every couple of weeks.
Sometimes the easiest thing to do is not the best thing to do.
It's nice in theory. Usually once these metrics exist, PMs and other managers get a hold of them, and they always seem to turn into hard deadlines or interpreted as predictions with high certainty.
Car 1 takes 15 hours to drive from A to B. Car 2 takes 10 hours.
Why is that? Well, car 1 has a 1.3l engine and took a wrong turn because the navigator can't read a map.
Car 2 stopped 3 hours in because it had a flat, but they got going again much faster.
The speedometer won't tell you any of those things. It can't tell you WHY you're going slowly (you're under-resourced or you have a problem that needs fixing) or if you're going the wrong direction.
That doesn't mean it's useless. It can still tell you if there's a problem, or if you appear to be doing well but you're not making any progress, maybe you're going the wrong way.
For starters, tasks require deep knowledge of how something will be done. In our industry, a lot of the work is discovered while executing some other work. What you get are tasks being created while other tasks are being worked on. This back flow of work would also have to be captured as part of the process with the additional metric of how well we plan our work and hold people accountable to that result.
Since software engineering is part art, industrial processes are inadequate in capturing the productivity of software engineering work. Instead, a lot of focus is on the ability to estimate overall completion dates, adherence to that schedule (often requiring evaluation of compromise and reduction of scope), and the impact of that work (both positive in a measurable way, and negative in code debt and incidents).
What's unfortunate is that a lot of infrastructure work focuses on improving engineering cadence. Without the process described in this article, measuring productivity gains from introducing a new tool or technology can at best be measured using adjacent metrics (e.g. compile time, deploy time).
I hope one day we can reach the level of engineering rigor prior to execution described in this article, but I still believe that the nature of software engineering is that there will be a non-trivial amount of immeasurable effort that doesn't fit the model described in the article.
I will say that “velocity” is an extremely helpful concept at the level of the individual and small teams. With my trams I challenge each engineer to try and batch work into 2-4 hour chunks, with each chunk being labeled “1 point.” Thus if we have an ideal week there are theoretically 10 points of “velocity.” We also try to batch up meetings etc. and discount the coordination overhead from the plausible output, so maybe the theoretical ideal is lowered to 8-9 points. We all discuss and agree on that based on whatever overhead we can’t eliminate.
Then each engineer does their best to “call their shot” every week and say how much work they’ll finish. At the end of the week we compare their prediction against reality, using a burn-up chart as a visual aid to the flow of work. There’s no penalty for getting it wrong, just applause for nailing it, and always a discussion of what worked and what could have gone better.
Following this habit, I have consistently seen teams get much better at forecasting work into the future accurately. This ability to have a steady velocity and make reasonably accurate forecasts about how long things may take is extremely valuable and helps us align and coordinate efforts across many teams.
Two questions, if you don't mind:
1. One of the biggest dangers of time-based estimation is the cultural danger around the "penalty for getting it wrong." It's got quite a lot of gravity to it, in my experience. Once there are time-based estimates for tiny bite-sized amounts of work, it's a hop-skip-jump to the toxicity of "holding people accountable for their commitments."
It sounds like you, as the team manager, are the one preventing this backslide. How do you think about maintaining a healthy culture around this when you go on vacation, get distracted, get promoted, etc?
2. What if a new discovery happens mid-week? Specifically, what if it's the kind of discovery that invalidates the current plan, so that finishing your "committed" work would be worthless (or actively negative) for the business?
It's easy to nail your estimations when you're operating at 1/3 your capacity, and you only ever meet but don't exceed the planned velocity.
I wonder if this also disincentives learning and innovation in the solution domain - because that would throw off all those estimates.
Also what you describe sounds... Like something I wouldn't want to be a part of. Breaking down smaller stories only to fold them into a bigger one?
(2) When something unexpected comes in you flag it and report it to everyone so everyone knows the expectations are invalidated. This is super important! I can't even tell you how much easier it is for a PM or other stakeholder to handle learning on Tuesday that the plan is not going to work than showing up for a check-in the next Monday expecting to see some deliverable and only then learning that it's late. The goal is to communicate expectation changes as close to real-time as possible, which builds trust and also helps the non developers see how volatile and unpredictable development can be so they learn to believe you when you say "the range is 10 to 30 weeks, and no I can't narrow it down any more than that."
Managers like to think, if an engineer can do 1 point in 4 hours, then they should be able to do 2 points in 8 hours. But that's not how it works.
The average office worker is only productive for about [3 hours a day](https://www.vouchercloud.com/resources/office-worker-product...). The upper bound is [4-5 hours a day](https://www.washingtonpost.com/lifestyle/wellness/productivi...). More than that is unsustainable and leads to burnout.
This is why people prefer remote work. Seeing coworkers in person is nice, for sure. But management still assumes people work 8 hours a day, meaning office workers spend most of their day trapped in the office pretending to work, which is a huge waste of time.
Also don't underestimate how many shallow tasks there are that somebody has to do to keep the lights on. We try and eliminate as many of these as we can, but it's just not cost effective to eliminate the entire long tail. So it's pretty common to have the morning focus session be true "deep work," and the afternoon session be a batch of minor stuff like small bug fixes, updating dependencies, running some tests, etc.
- 3-4 hours of deep work
- 1-2 hours communication, in the form of meetings, emails, writing wikis, etc
- 1-2 hours teamwork, like CRs, impromptu whiteboarding, pair programming, etc
- 1 hour misc, like coffee breaks, bathroom breaks, walking between meetings, etc
I think a lot of managers fail to account for everything else employees have to do, so “four hours of deep work per day” doesn’t sound right… even if it is. Of course, a lot of managers also just push their team into working more than 8 hours per day.
Two things are different to the XP approach. Firstly, you track expected work capacity at the level of individuals, rather than the whole team. Secondly, you use estimates of work capacity, rather than the "yesterday's weather" approach of assuming it will be the same as last week.
Do you think there are advantages to doing it your way?
Having never done XP I can't say if the differences are better, I can only say that this basic approach works pretty well :)
> Velocity metrics are as loathsome as the horoscope because neither provides any insight on why something went wrong or how to fix it.
Specifically, it assumes that there is that one metric out there than can answer the “why” and “how” questions.
In reality there is none, even your proposed CFD (which I use myself) doesn’t really answer them. Instead what these metrics do is answer the “what” question. Specifically, they signal to the team that something is wrong and they need to dig deeper to get to the bottom of it before arriving at a solution. In fact, one of your examples skips this very important step:
> A manager who wishes to make their team’s cycle times more uniform can try matching the rate at which tasks enter the system to the rate at which they leave.
How did the manager know that this was the correct solution. Perhaps they went further, asked questions, and maybe even consulted stakeholders instead of just looking at the CFD and go “Ah! I will control the arrival rate!”
This would’ve been the same with burnup/down charts provided that the user of these charts knows the probabilistic nature of these metrics.
In the end, it’s not the charts/metrics that matter the most but the person interpreting them.
Velocity seems like a simpler measure than a Network Queue, and both seem like they are valid measures since neither will have a linear relationship with the bottom line of the company.
The goal of the entire business is to tune all the parts of the business and seek out an efficient solution.
The theory of constraints as it is quoted in this article isn't advocating for rate matching everywhere in the business specifically. Rather it is making the statement that some inputs and processes in the business have a disproportionately large effect on the business outcomes. And you should weight management priority to the processes that affect your outcomes the most.
If you have a process (say deploys to production), that dramatically affects your business outcomes, in the sense that small changes to this process lead to large changes or even most of the changes in outcomes (e.g. revenue). Then most of the resources should be allocated to optimising that part of the business, since resources allocated anywhere else will be wasted, since the magnitude of the effects (even assuming they'll still be positive), will be dwarfed by your most sensitive inputs.
The theory of constraints preaches that after you have chosen your most important business outcomes (a strategic decision), you should measure how much the different inputs to your business affect the outputs of the business, and allocate resources to the most sensitive inputs.
Most big businesses do this very well. It's why you can find yourself in a team where doing more work or less work doesn't seem to matter, even though your direct manager is up your ass all the time.
The wikipedia link that you provided doesn't say this. It says:
"The kanban card is, in effect, a message that signals a depletion of product, parts, or inventory" https://en.wikipedia.org/wiki/Kanban#Kanban_(cards)
Which is quite different. The card is sent from the people who need the work to the people who deliver the work and not the other way around like you claim.
It's essentially a long term weather forecast. Don't plan your beach vacation for November; don't plan to launch Product Feature X before we could have it done. "Velocity" makes it sound like it's measuring your productivity now, but it's more of a general forecast of the future. And absolutely nobody should be looking at it but the PO.
- Points are a measure of time elapsed.
- Velocity between teams can be compared.
- Points between teams can be compared.
- Points are a unit of value.
- Velocity has meaning or utility outside of the team, scrum master, and product owner.
- Velocity measures or indicates individual productivity.
- Velocity needs improved.
Keep it simple, use it within your team to plan your next sprint and estimate when/what you can deliver.
And yet I still can hear some peanut gallery murmuring saying "Yeah, but, but..."
> Velocity metrics are as loathsome as the horoscope because neither provides any insight on why something went wrong or how to fix it.
This is such a weird statement to me. Of course it doesn't tell you why, it's just data. It's up to anyone that works with data to figure out what's going on.
Martin Fowler, who I had the pleasure to know back in the XP days, says:
“Agile Development is adaptive rather than predictive is people-oriented rather than process-oriented”
Reconcile that with agile as you understand and practice it - the entire point of measuring velocity is to be predictive. How on earth is that agile? How many process and planning oriented ceremonies do you go to? Does my empathetic listening approach sound more people oriented or OP’s “I’ll fire you if you aren’t burning down points” approach?
> Besides telling you that “velocity is low,” what else does a burndown chart reveal about your team’s bottlenecks, problems, and inefficiencies? Nothing.
I mean - it's good to know there's a problem (deadlines won't be met), right? It might not answer why, but it certainly allows you to ask why.
I'm also a little confused about what definition of velocity the article argues against while promoting a form of velocity called cycle time/avg cycle time; which seems to mesure how long tasks take to complete?
if your measure of velocity doesn't rate direction (i.e. 'quality of goal') as well as magnitude ('how quickly you are getting there'), at minimum it needs to be renamed
afaik most agile systems are pretty clear that cross team velocity is non-comparable. (if if teams are shuffled the velocity will change).
But, the one thing it's meant to be used for is estimates. After sufficient iterations of the loop a team should both have increasing velocity (due to knowledge of codebase), and increasing accuracy of estimates (both dialing into sprint point / effort tuning, and knowledge of codebase) .
afaik velocity was never meant to tell you about how to improve itself. Thats more for Retros. (retrospectives)
The velocity cannot keep on increasing forever :-)
Jokes aside, yes there is some asymptote of peak velocity that will reach. But keep in mind that some amount of effort can actually increase velocity in a net profitable way given long enough time windows... cue the table of automation from xkcd[1]. Imagine if we're playing with legos, and instead of using the smallest units I start to build with bigger blocks? I can build the same house by placing fewer blocks (analogy-- faster) .
Trying to model velocity as a "metric" also bakes in A LOT of assumptions. If the right cards aren't being created, or if the backlog isn't being prioritized correctly then what you are attempting to measure isn't even velocity (or at least not the specific velocity in the direction you ought to care about).
That was a fairly typical response, when I talked about having issues meeting deadlines. The Japanese did stuff like that, all the time, in hardware, but it didn't work, for software.
I have found that I am best off, trying to establish a structure that allows each engineer to work alone. That's difficult to do, when you have a complex, hybrid system. Usually, it meant lots of small tasks, with clear coupling and API specs.
If even one additional person is added, the whole thing slows down exponentially.
It’s predominantly recording point inflation over time.
Measuring things is hard. Most of the time when you measure something there will be a loss of information. Velocity, in the context of software engineering, is no exemption. However, this doesn't mean that it can't be a helpful signal.
I’ve been in “The room where it happened” a number of times, and this question just reveals that we have collectively forgotten what a burndown chart is for.
The point of a burndown chart is that management is asking the wrong questions, and you’re giving them something that looks like what they asked for but invites a bunch of uncomfortable questions later on. It’s insidious, in a useful way. What we’re trying to show is that wishy washy decisions - from management - keep moving the goal post and costing us hundreds of thousands of dollars. Potentially millions. It’s an up-management technique in the right - or even passingly competent - hands.
The mistake we made is in thinking that putting a mirror in front of somebody will lead them to improve. But what happens when you put a mirror in front of a narcissist? They thank you for your help getting that stray eyebrow hair to lay straight, and then change the subject.
I thought this would be about physics.
Is this currently being applied at your employer? Are you hiring? :-D