and no one is willing to say so.
On the reverse of course it makes sense to have estimates otherwise how should managers or product people decide what should be done next?
A big part of the issue is we don't have good knowledge how to estimate work. It's not a skill that is heavily invested in and is wrongly assumed as an inherent ability.
The biggest pet peeve is when I give an estimate such as "I don't currently see a reason why we can't deliver X by the end of next week, assuming no other priorities come up." Of course something more important comes up in the mean time and I get scolded in a large meeting with a statement such as "You promised me I'd have X by Friday!" — this is especially infuriating when it is the PM's other project that is the higher priority interruption.
In short it sounds like you don't have the power to say no - but actually you do.
Another thing that helped us is to answer with ranges instead of dates e.g., "this will take 1-3 weeks".
Of course, if you already have a toxic development culture like the one you describe, then they'd probably abuse more agile processes too.
Then I make more promises and estimates, playing the emotional card of "sunk cost" to convince upper management to continue funding the team rather than torpedo the project. Then we ship, and I quickly get everyone on my team promoted "because we shipped" before it becomes clear to upper management that we shipped something essentially worthless to the company. Because we're in a huge profitable company, everyone can jump ship to the next shiny thing, and whoever is left holding the potato on its way to deprecation just gets re-org'd when it gets canned.
I know it's a bullshit game. But I play it well, and all the software developers on my team get paid and promoted because I am so good at it.
Will you then argue that all new projects should avoid deadline dates, that no project should have anything other than an MVP and a iteration plan after that?
How do we change the expectations?
You sound wise and such in general, but perhaps your time would be better spent making a difference in how people believe work should be done at a place that actually allows brewing better practices instead of lauding bad ones.
This is exactly the point of the article: Organizations routinely fail to prevent untenable projects from moving forward.
Why not work on something so valuable that even if it takes twice as long as the best estimate, with twice the headcount working on it, it would still be worth it?
As a a manager, I feel that helping to find projects that have that amount of value is an important part of my job, so that if I am fighting for a project my team's working on, I'm actually in the right from a business case point of view, not simply playing some BS political game.
If there are no projects that fit that value filter, then that's a completely different organizational problem.
On one level I have to take my hat off to you because you sound like exactly the kind of manager - one who has your team's back - that a lot of people would want to work for.
On another level you're part of the problem: you're wasting organisational resources that could be better and more profitably used elsewhere.
Still, needs must I suppose, because the real fault here - as the article itself hints - is in the determination of what the most valuable use of everyone's time is. I.e., if projects are going to be late (and they are) then you ought only to deploy teams to work on the ones with a putative 10x ROI.
But there was one occasion, where a product launch had been announced by a C*O before the product was finished, and the engineers were very clear that this launch date was impossible. So there was a lot of drama and bad feelings, and the engineering team eventually said, "okay, let's make a detailed estimate so we can be certain". The estimate came in at three weeks development time, one week after the announced launch date.
One week of the estimate was meant for testing and fixing problems that are found in testing. The business experts immediately jumped on this: "Testing and fixing unplanned errors won't be a big deal and certainly not a whole week. We'll skip that so we can get it done in two weeks".
Thankfully, this is retail banking, so releasing an obviously broken product flies even less than missing the deadline. The product was eventually launched uneventfully, one week after the detailed estimate and two weeks after the announced launch date. All in all a success.
* If you just don't know how long something is going to take, and you need a date, then you're late to doing the work to know how long it will take.
* If you do know how long something is going to take, and it's longer than a externally imposed date, then you know exactly how late you are.
This is partly just playing with semantics, but it's also a way of talking around different views of the problem, in this case emphasizing that if dates are a feature, reasonable estimates are themselves a project. If the estimate work hasn't been properly completed and understood before dates are given, obviously, the larger project is by definition already late because there's work that needed to be done before beginning and wasn't.
And I think if you take this view, it's easier to see that it's a project or team management problem rather than an engineering productivity problem.
The problems start when you need to give an estimate before even being allowed to start working on something.
That means building a simple verison first that still fulfills the purpose while not being perfect or polished. Then iteratively improve it to what you and the customer ideally want.
That way, if the deadline comes before you're done, you at least have a working version that fulfills the basic need and allows the kind of workflow that's necessary, even if it still has some rough edges or even gaps in rarely used areas.
Of course this approach requires you to write your contracts in a way that doesn't tie you down too strictly on how exactly things will be solved.
So the customer can have assurances that they aint gonna be bogged down with sunk costs. The developer can have more frequent deliveries.
Please elaborate! :)
Let me first acknowledge that there my thoughts are a mixture of the types of projects that I've been working on, my own personality and my own path dependent experience. In particular, my way is intentionally short sighted and won't work for a project plan that takes five years. Also, for sure, if the sales team/ceo is wrong and the project isn't worth doing, then doing it "lazily" won't fix that. But with that out of the way, here's what I mean.
The engineering team has it's own values and priorities and they just don't have anything to do with what really matters to customers. In the end, everyone brings in emotional baggage from the last project they worked on, and that's what's important until new information comes in from customers and the sales team.
So what I shoot for is taking the requirements that we have and producing a long range design that will meet those and little more, and then producing features as quickly as possible and presenting it to the customer (or the customer's advocate).
Why does it limit scope? Because the deadline doesn't move! To meet the most important requirements in a way that actually sells the product, lower priorities are are cut or reframed.
only if those doing the promising knows what's within bounds and what's wildly off bounds!
If the sales person makes wild promises because his commission depends on making the sale, but not the delivery, then the whole organization's interests are not aligned properly, thus fail.
If the salesman's commission comes from successful delivery, then he can bring in an engineer who can at least balance the promises. Who knows, may be different departments can cross-contaminate, leading to a better, more well rounded organization.
Now he’s amending his opinion to state that estimation is not really the core issue. The core issue is that projects are doomed from the start by mismanagement. Issues such as: racing to catch up with a competitor (giving the s/w group an impossible deadline), or starting a project whose business value is so low that minor time and cost overruns reduce the actual value to below zero (at which point people start blame shifting to try and hold on to their jobs).
Instead, we systematically underestimate.