Also, a job on his home is exactly what it should not be compared with, since that's the sort of repeatable job that can be estimated.
To some degree and depends on the job. Start opening up walls in an old house and tell me how cookie cutter it is. When I used to run shipyard jobs on offshore drilling rigs, I can tell you that every single job ran into all sorts of problems that were individually not predictable.
It's also just nonsense to think that most software is this uniquely open-ended snowflake. Most engineering jobs require problem-solving that can be affected by all sorts of things in the physical world--plus you sometimes just have to build and try out designs and maybe they work, maybe they need to be refined a bit, and sometimes you have to toss them out and start again from scratch.
I see people on here post construction projects as the poster child for things that can be estimated with traditional project management techniques. This is wrong because most construction projects come in over cost and over schedule.
This can be because materials costs change, labor costs change, the soil at your site may not be what you expected, weather impacts, and stakeholders may demand something different after the project is estimated.
Although these are unknown quantities, these factors existence are known. Whereas in software, you’re more likely to encounter unknown unknowns. These are factors that nobody thinks are even possibilities until you run into them because nobody has done anything similar before. An example from my experience is when I was working on implementing a marching cubes algorithm on a GPU, we did research to make sure that you could do it. But when it came time to implement, we found that the cube data was expressed as a set of faces, and not a set of vertexes and no efficient algorithm existed for that expression of the data that could be mapped to a single instruction multiple data paradigm. We could not preprocess the data to convert it to vertexes because it would not offer any speed up if we did. To implement the acceleration we wanted, you’d have to change how the data was stored in the application. Which is a very different project with very different timescales. We ended up abandoning the effort and moved to other work. This happened because we made a discovery that no one could have predicted, in the implementation, that had an unknown impact.
In physical projects, you might find there are a lot of unknown things that can happen, that you may not predict. You can certainly reduce this by bringing in experts to a project, people who have seen much more than you. Certainly there is still potential for black swan events. But these are rare. Where as in software, the experts you may bring on may not necessarily know more than you, you are frequently your own expert. Black swan events in software projects are common.
But they were surely predictable in aggregate, no? There's only so much volume in the house and a volume will take a bounded amount of time to deal with, even if the bound is very large. An oceanic vessel has a maximum amount of mass that can be added or removed, and every unit mass will take a bounded amount of time.
This isn't the case for development. There is a bound to how much code can possibly go in a project, but there is no bound for how much development work goes into any unit of code.
Only in the sense of jobs seem to take at least twice as long as they "ought" to. The volume of the vessel has very little to do with it. In a refit, you know some things going in--which may indeed be relatively straightforward (so long as the right parts arrive on time)--but for bigger jobs lots of things crop up including resource conflicts and parts that need to be ordered.
If software projects added more slack, they'd be more likely to be completed on time too. There's nothing magical about software. And, in fact, software doesn't need to deal with a lot of the quirks of physical systems.
No, just all software that provides any sort of tangible benefit to anybody. I’m sure you could estimate fairly accurately how long it would take to implement quicksort - but nobody’s going to ask you that.
I know many developers enjoy the art of software development and want to continually improve it, but that isn’t always the task you are being paid to do.
There are many ways to remodel a bathroom. Some more optimal than others.
Now imagine your plumber tells you he discovered a better layout for the plumbing just after finishing his work, then he enthusiastically asks if he could tear everything out and start over (and pay him for it).
Vendors and even open-source libraries are often liars or just missing what seems like it ought to be obvious functionality. Sometimes tools are just broken. I once was assigned a Java AWS Lambda project, which, great, supported, right, says so right on the website? Except half the CLI tool functionality just didn't work, at the time, for Java projects, and I mean the official tools. It was alpha-tier supported at best. Usable, sorta, kinda, but you're gonna lose some time un-fucking the situation, and the whole thing would be slower to develop than it needed to be. You'd not know that unless you went and did a focused search on their issue tracker(s), which isn't really reasonable to do for all the stuff you might use in a typical project (some of which you might not even know you need until you've started work). "Great, so now you know that for estimating next time, if you get assigned a task exactly like that one" well, no, not really, because that was like four years ago and the situation's probably totally different now (maybe better, maybe worse, who knows?)
That's why I think giving 4-8 weeks of development time with some basic t-shirt sizing or pointing for tasks and just seeing how it's going is better, before estimating. I think if you can't afford that, before getting the estimate, you can't afford a half-decent estimate at all and just need to deal with the fact you're going to get a terrible estimate with an absolutely huge range ("six to eighteen months"). Which, fine to ask for that up front then refine it after some dev time! It just needs to be clear and expected that those early estimates are going to have some wild variance—but they can definitely tell you "is this more a 1-year project, or more a 5-year project?" which does have some real value.
It's not strictly correct that most software is open-ended research. Most economically valuable software is open-ended research, because otherwise a solution already exists and you don't need to pay a software developer to write it.
Software is a somewhat unique industry in having zero cost of replication. The only other industry like this is creative & artistic pursuits (writing, artwork, music, film, etc.). That industry is also notoriously difficult to estimate and tends to be an area where the the greatest works of art are done when they're done. (How long have people been waiting for George R. R. Martin to finish A Song of Ice and Fire?)