With that said I totally get why designers hate Agile - it forces them to make design decisions they're not ready to make and once there's functional software in people's hands everyone groans when design suggests non-feature changes.
In my experience agile/scrum just adds to the politics within the org. It’s a bloody religion. Agile adds in agile zealots vs the infidels.
All of those dumb politic things get no pushback until the bitter end of the project when everyone is trying desperately to ship it before the board gets pissed off enough to start rolling heads. Then you get some death bed conversions that don't really change the fact that the real change was needed six months ago and now it's all going to be bailing wire and masking tape.
The fast feedback loop makes the bullshit quite apparent early on. And if you're very careful, you can arrange for some of the features that never should have been agreed to in the first place to be at the end of the line, where you can say, "look, we can ship what we have now, which does most of what most people wanted it to do, or we can keep farting around with this for another three months".
A big part of where this breaks down is that you have two groups, one of which hates to change and one of which has had to become more comfortable/resigned to it, and if you put them both in the same pressure cooker, group A is still going to evolve slower than group B, except now group B has concrete evidence about what a group of children group A is. Now both sides are looking in the mirror and neither of them like what they see.
The problem with knowledge is that once you have it, it's hard to go back, even if you don't like what you know. So I'm not sure what the road out of this is, other than the ones that sort of already exist, like OSS, which creates little pockets of what should be sanity (but often are just a different flavor of crazy).
From the Wikipedia article:
"The one-year delivery target was nearly achieved..."
"A few months after this first launch, the project's customer representative—a key role in the Extreme Programming methodology—quit due to burnout and stress, and couldn't be replaced."
"The plan was to roll out the system to different payroll 'populations' in stages, but C3 never managed to make another release despite two more years' development."
"Frank Gerhardt, a manager at the company, announced to the XP conference in 2000 that DaimlerChrysler had de facto banned XP after shutting down C3..."
Most engineering disciplines spend a lot of time for creating designs and blue prints before proceeding with the implementation phase. The mistake is assuming that software requires that too. It requires a little bit of thinking for sure. But fundamentally, the ultimate blueprint for software is executable. Your blueprint isn't finished until it runs. The process of producing the working blueprint is the whole point of a software project. You produce an executable specification in some programming language. The process of translating that to a working system (aka. compiling) is automated and does not require any human input typically. So, you do that very often starting on day 0 of the project. Until you are satisfied with the results. And why stop there? Many software products continue after their first release.
If you are building a nuclear plant (or any similarly large civil engineering project), producing the blueprints takes many years. In fact, nuclear plants are infamous for lengthy delays during that phase. Almost like a big software project. Unexpected things happen. Which then require changes to the blue prints. And until you have those, you're haggling with authorities and other stakeholders over permits, requirements, safety, and what not. It's fundamentally an iterative process that doesn't respect deadlines. Like software. Of course building the thing is also not without challenges but there's typically no room for a lot of improvisation during that phase. So, it's relatively easy and more predictable.
You can't really plan a hardware projects until after you have the blueprints. Hardware is just like software in that sense. The difference is that with software you are done once you have those. Planning the deployment (or customer implementation as it is sometimes referred to) is relatively straightforward: take the thing, install it, and run it.
EDIT: You didn't do Waterfall, just to be clear. You used prototypes and feedback along the way which Waterfall explicitly excludes. So why do you insist you participated in Waterfall projects?
I'd say this is the case more often than not.
In order to advocate for Waterfall instead of any Agile methodology during the last decade or so one has to be overwhelmingly blindfolded by the religion. And these kind of folk tend to be quite strict in their faith. As in: "we had enough time to discuss things and think it through and now it's time to stick to the decisions we made".
And yes, I have witnesses just that quite a few times.
I know a software company that strictly uses classic Waterfall even to this day. A minor feature that should typically iterate in production for like 3 or 4 times and be done takes them about a year to implement and still never deliver what the customer wanted. And I'm not exaggerating. (The company prints free money from another line of business).
do you think its possible that we all said 'yep, our planning process is faultless, all we have to do is wait 18 months and it'll come out exactly as we conceived'. how blind we all must have been
Sadly, though, it is in fact a real thing. I've seen several projects fail spectacularly. One of the best failures was "only" a year late. They also learned not to go 6 months into development without integration (believing their plan was sound) and skipping testing for that entire time. It was either brainless or insane, and it took a lot of threats from customers for the PM to finally learn that, maybe, a plan made a year or more out and committed to wholeheartedly is stupid.
Yes. I know from direct experience.
> how blind we all must have been
Yep. The industry really was that dumb. Much of it still is.
On the one hand I try to remind myself of this when I start feeling like nothing is going right on the project. We are less dumb, and I'm applying the same(?) energy to things that matter less.
On the other hand, telling people for 15 years that we should be doing X before people start reliably doing it is fucking exhausting. If I thought about it, I probably avoid reading about things that are too 'new' today because I'll only get to use them in my dreams.
People in startup culture have never actually seen what waterfall can be like. Waterfall as first 'proposed' was meant to be a strawman used to show how not to do things and some managers thought it sounded pretty good.
Then the finger pointing happened.
Ops said, "Our servers aren't slow or missconfigured, QA should have done load tests before approving it. "
QA said, "The code we got was shit and buggy, it was all we could do to get all the breaking changes found before ship date!"
Devs said, "Product kept changing the design!"
Product said, "Sales sold something ill-defined and over promised!"
Sales said, "Everything we do isn't sellable! We needed these specs to compete! The reason the customers left was that Devs took too long to develop! Product couldn't write requirements! QA didn't test!".
So a solution was drawn. Product would get sales to get their sales-engineers to pre-qualify opportunities and sign off on a pre-qualification spec. Product would accept that, and once accepting it they would own it.
Product would take the Pre-Qualification doc, and produce a detailed requirements doc. Including timing estimates for interactions. They would present that to eng, and eng would sign off on it. Once they sign off, they wouldn't allow any changes to it. Product and Eng would need to complete a change request for every change to the requirements.
Eng would take the requirements and produce a detailed design doc. Then they would setup a time estimate that they would be held to. They would produce a machine spec sheet they would give to Ops to get the hardware ready. Once they complete eng, they would present the completed code to QA.
QA would have a test plan based on requirements only after they would accept the code from Devs. Once QA accepted it, they had a timeline to complete testing. When they have reached a low bug count, (Having a high number of bugs initially should have been caught by the pre-QA Test plan), they would sign off on the quality doc and hand code to Ops.
Ops would deploy the code, if the product didn't perform, it would be assumed at this point its the hardware config and thus ops issue.
All of this would be designed that the team wouldn't be "at fault" if they got the next team to accept the code, and every team would make sure everything was perfect before they accepted it. To keep deadlines, minimum times for doc creation and acceptance would be created. The only variable would be dev/qa time.
The process would be fool-proof, guaranteed that nothing could be produced that would cause another fuckup. Typically the first major project to go through this process would be the last. Either the company died before it would be completed, or the process destroyed itself in such a massive flame/backstabbing fest that even the CEO would get fired.
Waterfall incorporates feedback. You must be thinking of something else.
But as things become more unpredictable, you need a process that can respond fast to changes.