Dear Agile, I’m Tired of Pretending (2018)

And it turns out that it's not unusual for software developers (and consequently software development organizations) to spend more time in the we don't know what we're doing phase than the we do know what we're doing phase. That sucks from a management point of view, but no amount of process is going to make the problem go away. The best you can do is manage the risk. We don't need good processes or methodologies nearly so badly as we just need management enlightenment about how software development actually works.

I agree with most of the article. Specially the keep-learning part.

All Agile did was put software development teams unfairly under a microscope.

I believe Agile has been tremendously beneficial for the industry globally, especially in some subtle ways. For example, Agile says you have to communicate a lot, if you want to get software done. Here is the subtlety: if, today, you stop telling the average programmer to communicate, they will stop and go back to silo mode, "naturally". At least some of them. If you think about software at the industry scale, you have to think about a wide population which require processes.

I was once working in a nice little company where, one day, they introduced agile. It was so much beneficial. Before, the bosses thought that talking was simply a lost of productivity. You're a programmer right? So code, don't talk. So we would develop software without talking to each other. After we had daily meetings, scrums things and stuff. Our velocity has sky-rocketed.

You have to realise that before Agile, a fair portion of all software development projects that were started would simply bust and never get shipped. The code is a complete monster or the budget is nuked. When I started in the company I just mentioned, I started working on a codebase that was the worse code I've ever seen in my life. You would touch one line and everything would stop working. It was a condensed piece of spaghetti with hacks on top of hacks on top of hacks. Software architecture? That requires some talking and thinking, forget about that, not permitted.

Now I agree with you, agile is not much useful for a hacker who consistently get software done and understand deeply what's happening.

edit: not that I'm an agile guru or anything. Actually I only know the thing superficially. I show up at the meetings, when I'm ask how hard something is, I answer, then I mess around with my office friends. Still I can appreciate it works much better than any process the bosses can come up with.

> both Agile and Waterfall are focused on building. Design is about validating.

or

> So…what’s the way out? It’s a smart focus on clear outcomes, not output, with roadmapped outcomes replacing planned milestones, with trusted product teams, not project teams, empowered to vet assumptions and discover the minimal path to value.

Satire? Enough words to fill a hot air balloon or two. "Smart" focus, "clear" outcomes, "outcomes" vs. "milestones"(?), "trusted" product teams (as opposed to... what?), "empowered" etc.?

Far more interesting than the contents of many links that make it to the HN homepage is discovering the fact that they do so.

The problem is usually solved in three important ways:

1) making the deliverables as vague as humanly possible

2) promising experiments and not results

3) applying for grants for work that you have already completed, and using the money to work on the next grant's project

The problem of having deadlines that are too far out is mitigated somewhat by the pressure to publish. Publishing has fixed deadlines but somewhat flexible deliverables.

Your goal should never be to become "more agile". You should be looking to improve efficiency, to make sure you are building the right thing, to improve communication so that everyone is on the same page, etc. "Agile" is never going to magically improve anything.

What to do if you're on an Agile team? There's value in the Lean Methodology. Remember, the whole point was to deliver software your customer needs to fulfill their business objectives. Continually assess your Agile practice and see if what you're doing still makes sense for the project you're working on right now and for the current stage that project is in. This continual assessment with a focus on the end goal of delivering software can go a long way to addressing Agile's modern ills.

Not everyone. Some are very successful with it. The problem is that everybody wants to copy that success and tries to cargo cult themselves through it without understanding it.

"let’s stop pretending Agile was some sort of cure all."

That's certainly good advice. Agile is more of an attitude than a solution, and attitude is not going to solve all your problems. Even if you are as Agile as you can be, you still need to solve your problems. It's just that being Agile might make you more flexible and effective at solving those problems.

And Lean is certainly more comprehensive than Agile. And I don't think they contradict each other. In fact, they're very similar; they're both about empowering the people who are doing the actual work.

"management thinks the focus should still be scope and deadlines and efficiency, ignoring that the deadlines are arbitrary and the requested time estimates are a form of waste."

Then management is not Agile and hasn't properly learned what it is. That is sadly common. Agile is not magically going to solve bad management, but Agile does emphasize keeping management at a distance and empowering developers.

"Agile actually tends to mask the core problem, which is a systemic, bidirectional lack of vertical trust."

Does it mask that? I think it exposes it. Without that trust, Agile cannot work. Agile insists on that trust. I suspect that every case of broken Agile is caused by a lack of trust. Fix the lack of trust (or use a system that works without trust if you absolutely must).

"Did you know story points were actually invented to obscure time and help alleviate this problem? That backfired too, didn’t it?"

Yes I did and no it didn't. Every single team I've worked in over the past 6 years knew this and kept repeating it every time someone comes along and tries to match story points to actual hours.

This first half of the article suggests the author has mostly worked in very toxic management cultures.

On the other hand, I've also been in orgs that were very successful in developing software in a way that we would recognise as being agile. We just didn't make a big deal out of it. And we didn't do standups (most of the time), or 2 week sprints, or retrospectives. We didn't even pair consistently: we split up on trivial stuff, paired on hard stuff, as needed and available.

In one org we started without TDD, and even without having much in the way of automated tests at all. But when we discovered testing and TDD, boy did we ever embrace it. Not because anyone told us to (mid-to-late 90s, so no Agile Manifesto), but because the difference for us developers was night and day.

So develop software. Concentrate on what helps you develop software. In my experience, you will probably discover something recognisably agile all by yourself. So when you discover that, help yourself to what helps you. Don't make a big deal out of it.

One thing I see a lot in the comments is about time estimates and the amount of pressure that that brings and the rarity in which it works.

In my teams we NEVER talk about time of delivery with Engineering. I've always been trained to think of pointing as an estimate of "complexity" (e.g. 1 point is roughly the complexity associated with a copy change).

Then, by getting a track record of historical points completed in a week, we can transform our way from feature => complexity => time.

Now I believe that Agile was started with good intentions, but today it is much more a motte-and-bailey doctrine. The motte being "Agile is the values in the Agile Manifesto". The bailey being "Agile is Scrum and SAFe, and all these meetings, ceremonies, and process are a necessary component of Agile". And I think this is why people fall into no-true-Agile discussions and eventually end up disappointed with something that showed a lot of promise. Agile started off a plea from programmers to Corporate, and it caught Corporate's ear. But once in Corporate's possession, it was deployed to serve Corporate ends only.

Among other things he speaks about the simplicity of the Manifesto, about the cultish movement and consulting services which rose around it and the cultish, distorted, complex vortex of BS that has formed around a super simple concept.

https://m.youtube.com/watch?v=a-BOSpxYJ9M

Software development is solutions development; but for a large subclass of problems, custom-built software that automates the tedious, predictable, and repetitive parts of the solution has been and always will be a competitive way to solve them. It's synecdoche, a metonymy for "software development [and other assorted disciplines of problem-solving]".

Software is the hammer in the toolbox. Things that look like nails get hammered. The hammer is so useful to organizations that have many nail-like objects sticking up, that handypersons who only have a hammer in their toolbox can still work continuously, and may never need to do anything else but hammer things. This may become common enough that folks who also keep vice grips, screwdrivers, hand saws, duct tape, and penetrating oil in their toolboxes are forbidden from using those tools. They were hired for the hammer; they should use the hammer. And what's all this nonsense about non-hammer tools, anyway? We've never needed anything but hammers here, and that's the way we like it.

Agile is about advising organizations with strict rules and style guides for nail-hammering to loosen up and let the hammerers determine how to wield their own hammers, and possibly also choose the length and width of the nails.

Despite the amount I actually get paid, working upstream of the problem far enough to eliminate both the nail and the hammer--possibly using a more appropriate tool--is apparently above my pay grade. Agile never fixed that. It never even suggested that a non-hammer world exists. Even the CTO position might as well be chief hammering officer. The 'T' restricts the domain of authority. An organization that is agile instead of just Agile needs to have someone around that can meaningfully restructure the company itself, such that the assumptions already built in to its organizational hierarchy are not inevitably replicated in the processes of all of its subdivisions. That's why Agile never fixed anything. The CEOs still kept telling their CTOs to take charge of all the hammers and pound something out. Those companies founded to be Agile put different assumptions into their organization. When that changes, going through the motions of Agile won't help them any more.

Unfortunately my tasks tend to be nebulous. They involve investigating very large potential systems and breaking out smaller tasks, many of which won’t be fully understood till we’re in the trenches. How do I cost the time it takes to learn how Spanner will handle my design and update architecture accordingly? How do I cost “interrupt driven” work like getting buyoff on a given architecture? One of my OKRs was literally “Create a security model for a new foundational technology that is acceptable to all integrators, follows Google Cloud best practices, and can be represented in Kubernetes.” How would you break that into (bi)weekly sprints? You don’t even know what work you’ll have to do until you go to a committee of stakeholders and get told where you have to go back to the drawing board.

I blame the ambiguous manifesto.

Agile is not and has never been a cure-all. Scrum doesn't solve any problems. It's supposed to reveal the problems that exist and make them apparent to anyone looking at the scrumboard. If you don't change your team dynamic and just start looking at a JIRA board every day in 30 minute "standup" meetings, you're not agile. You're just cargo culting.

I'm tired of people blaming "agile" -- which is mainly about self-organizing teams and adapting quickly to changing circumstances, for what's really a management failure.

Here's the thing. Corporations are hierarchical. Human beings are hierarchical. We're also communal, but start putting us into larger and larger group situations where we're facing competition, stress, and we must engage in concerted effort, and we go into hierarchical mode.

Why does everything tend to go back to waterfall? It's because that's how management likes it, for reasons which go down to basic human nature in the kinds of environments technology companies create. Heck, even the financial markets demand fictional planning and forecasts of the unknowable future. Bosses want to be able to make plans.

One thing which Waterscrumfall has achieved, is that the iterations are much faster. That even fits the "next quarter" mindset of the financial markets.

It is kinda funny. For me the first line has always been:

> Individuals and interactions over processes and tools

And when I am asked to explain Agile, that is always at the core of my message. Form a team, let them do their work and start learning how they can improve how they do it.

From my perspective, the primary role of a (software development) Project Manager/Scrum Master is to create an environment where the team can optimize itself while not drifting into personal conflicts (create a healthy environment for collaboration and feedback). Everything else, like planning/risk management/change management/stakeholder management/reporting/presenting come afterward.

Those other things are important too (and should not be skipped completely), but if the team doesn't work, most of the other tasks create little value.

"Agile" is a lie that before 2001 nobody at all had ever done iterative software development, and all software process ideas that came after 2001 are by definition "agile."

"Agile", as with any other buzzword, is not for people who actually understand the details of the change; it's for someone who needs a handle onto concepts that would fly over their heads without a buzzword. Lets stop litigating the term, and move on.

So how do you trust someone who would outsource your position if they believed it would save money and reduce their work load? Management and employees have an inherently adversarial relationship that is poisonous to trust. I think it can still develop at times, but under our current culture where everyone knows someone whose job was made redundant with little recourse for their employment, you might as well be asking for a coconut to grow in an active lava flow.

I've seen software development done very poorly and done very well, sailing under a variety of flags. But on an eyeballing there are far more bad experiences than good that get labeled "Agile".

Whether this is because of causal differences or because it's easy to affix any label to anything (and most experiences suck regardless of the label) is hard to tell without a control-group universe were the Agile Manifesto was never written.

If anyone needs me, I'll be counting to twenty.

1. Developers size development of X and says it will take roughly 12 months.

2. Management says the rest of the organization is going into system test in 9 months and needs X finished by then. Couldn't there be some way of speeding up the development of X without impacting the rest of the organization?

3. X's developers say no, not really.

Now management tries one or more of the following:

-- find someone else to size the work that X's developers will be expected to do

-- hire some outside "expert" to size X

-- hire contract programmers that claim to be experts in X that will join the X team

-- suggest reduced functionality for X

-- propose some modified "streamlined" component testing for X

-- promise to provide X's required dependencies ahead of their scheduled completion dates so X's development won't be gated by these requirements

Now jump back to step 1. Repeat until X's team relents and says that maybe, just perhaps they might be able to finish X in 9 months if everything goes perfectly even though the company has never done X before

This is now passed up the management chain and it becomes part of the official overall system's schedule that planning, marketing, sales, finance, etc. is depending on.

Iterating over all of these estimates is a big waste of time. Management has already decided that they want X in the system and will keep pushing until an estimate comes back that they accept.

If X's team is lucky, some other team will fail to make their estimated completion date and X will end up with the 12 months it might take to finish X. Every team in the organization is hoping that it will be a different team that first has to admit that they will not make their date. This causes a kind of game of chicken where every manager is unwilling to be the first to admit that their team is going to miss it date. Consequently, every date in the organization cannot be relied upon. The distributed file system needs the TCP stack to be finished on time, but the TCP stack needs the ethernet device drivers to be finished on time and no one will admit slippage of any dates first.

Managers often tell me that my process is chaotic. They want more visibility and deterministic completion dates. We adopt Agile process. Now we have long sprint meetings. I think we get less done.

> So find a good booklist. Follow some good blogs. Here’s a start: If you haven’t read Sense & Respond, Lean Enterprise, A Seat at the Table, and Everyone Is a Change Agent, I suggest you do so pronto. Your leaders too.

I'd add Peopleware to that list.

> Start reading posts by John Cutler, Melissa Perri, Bob Marshall, Allen Holub, Laura Klein, Erika Hall, Neil Killick, and branch out from there.

Anyone care to add their own handful of books and bloggers to these recommendations?

But we won't. That's why snake oil salesmen were so effective: we want to believe in the cure-all. No matter how often you chastise, people'll keep buying it, because they're aspirational. They want there to be greener grass on the other side, and they're willing to try anything to find it.

The business doesn't actually care about Agile at all. It just has heard of the term and realized it has something to do with getting work done better. So leadership will say things like "We're an Agile organization now", not having the faintest idea what that means. A snake oil salesman doesn't need to know how snake oil works; he only needs to know what it promises, and how to sell it.

You can have the perfect software development methodology and people will still do it wrong. But if everyone has the same goal in mind, is highly motivated, and is willing to cross aisles to work with other people to do the best job possible, a methodology will naturally arise. Self-motivating humans can get some crazy shit done. The best working organizations I've ever seen followed no methodology. They just did what was needed to keep a site running well and making money, and it was kind of wonderous.

Basically, my team consists of some devops engineers mostly working on maintaining and extending the config management, and some other technical consultants who deliver and execute projects, ideally fully standardized.

For the standard delivery projects, it's extremely valuable to hammer the scope down into a precisely known constant and track the time spent to implement specific parts, as well as the delays incurred for different reasons. This allows for a strong, deliberate optimization of execution time with a measurable, predictable business impact as well as a reliable estimation of these tasks.

Discovery projects on the other hand? Yeah. We're pretty good at giving a lower bound for the time necessary by tracking similar tasks in a job. Extending a database entity is going to take a roughly similar time each time. Packaging, downloading and extracting binaries isn't going to differ that much in time from the last 3 times we did that. If you can identify a bunch of similar tasks you've done some time ago already, you can give a lower bound - "The tasks we consider familiar will take about 2 weeks".

But after that? Who knows. At that point we're rather estimating by asking: When do you stop feeling silly about it? Do you need a year to do this? A month? Two weeks? A week? A day? So usually we hand our project managers something like "This will take 2 weeks we know, plus 3ish weeks we don't know".

Principles behind the Agile Manifesto

We follow these principles:

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.

Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

Business people and developers must work together daily throughout the project.

Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

Working software is the primary measure of progress.

Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

Continuous attention to technical excellence and good design enhances agility.

Simplicity--the art of maximizing the amount of work not done--is essential.

The best architectures, requirements, and designs emerge from self-organizing teams.

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

But I would like to see an organization that only had the team lead give estimations. This would remove the temptation on individual team members to size too large and fill the time, while also keeping management happy that they're receiving estimates for tasks they requested. Measuring throughput would immediately become a lot more natural since team members wouldn't fall into the trap of "When a measure becomes a target, it ceases to be a good measure."

I was actually just rejected from a job I applied to since I voiced a few small concerns with agile/scrum in the interview. My friend that joined that company recently reported he was doing maybe 10% of the work he and I were doing before he left my team. Even though I aced the technical portion they didn't want to risk hiring me since I asked their willingness to flex the rules of scrum.

A very good engineer within my organization said to me, "Our job is to get something in front of them quickly so they can figure out what they don't want." Our successful projects have been ones in which we kept to this maxim.

I think this is what the author was trying to say but I had a hard time following his writing style.

Instead of thinking you can write a perfect requirements document, then can just design and build it (waterfall) doesn't work.

The idea that you can have any sort of meaningful estimate even when you don't have perfect requirements is insane. Having an estimate with perfect requirements doesn't even work.

The most effectivly analogy I've used lately is: "Take your car to a mechanic and say "My car is making a funny noise, how long will it take to fix?" and see what kind of response you get"

Agile is being blamed for problems that are really organizational, company-wide issues that a software team has to deal with because they now are central to how everything runs. The author's suggest to read more general business/organizational management type books is apropos, because the overall goal is agility of the entire organization, not just the software team.

1. Developers need to communicate with each other. A lot. This takes time. The amount of time goes up non-linearly with the size of the team. (The Mythical Man-Month claims a scaling law of n^2.)

2. Adding process + documentation of various kinds can turn that from unbounded time spent speaking to a large but bounded written effort.

3. Nobody is good at predicting schedules.

4. Good news travels, bad news doesn't (until the last moment).

The traditional solution was to solve #1 with #2, put a lot of effort into schedules for #3, and get blindsided over and over again by #4. The average result was that the average delivered software project took over 2x what was estimated, cost over 2x, and was delivered with under half the promised features. But, no matter how delayed, usually didn't start slipping the promised deadline until about 2 weeks before the first official release deadline. And these were the success stories, since most software projects never really got delivered.

What Agile opened up was the perspective that there were other approaches possible. For example we can solve #1 by having small teams, which makes #2 unnecessary, solve #3 by making projects small enough to not need good prediction, which creates a feedback loop to avoid #4. And that is not the only valid tradeoff.

But this is not a one size fits all. Real world data shows that a team hits peak throughput at 5-8 people, declines, and doesn't get back to the same overall throughput until 20+. (After which it is almost linear, but with much lower individual productivity.) The teams as drawn on the org chart don't matter. The teams as they actually interact, do.

If you understand your options, and understand the tradeoffs, you can figure out how to customize the right solution to the people/organization. If you don't understand the tradeoffs, the best that you can do is blindly copy what worked somewhere else and pray that you got it right. And if it turns out that you didn't, develop superstitions about what you should have done instead.

Did anyone - who has seen shiney new panaceas come and go - believe it was a cure all? What it is, like MVP, is a way to mitigate risk. It doesn't eliminate risk. It can only mitigate it.

Furthermore, it still boils down to team and people. No tool will save a dysfunctional team. The key isn't the tool. The key is the key agreeing on the tool, and sticking to it. That is, tool is not a proxy for deep and solid team work.

To the extent your work isn't repetitive, the only job process can do is set a cadence on communication & decisions. Yes 2-week scrum does this but in my experience it distracts companies from the real goal of management -- attaching subject matter experts to projects, i.e. concentrating time & resources towards a goal.

Process can get rid of bad people and share around your best people, but unless you have an assembly line, beware of any other claim.

Heres's a good one by Dave Thomas. A quote: "Agile is an adjective not a noun". https://www.youtube.com/watch?v=a-BOSpxYJ9M

As a lot of people have already said, the problems arise when managers expect agile to give them software on a timeline.

Yet folks expect Agile to solve everything for some reason, and forget all the other issues around bags of meat working together. Those continue to need handling and still get out of control at times.

Perhsps the qustion we ask in projects shouldn't be "how long will this take?" but: "How long do you want to spend on this problem?".

Are other ladders in an organization not subject to performance review?! Managers? PMs?

https://en.wikipedia.org/wiki/No_Silver_Bullet

Beautiful

This was Ken Schwaber, co-creator of Scrum. 10 years ago. Ten years after, things did not get a lot better for big corporations that were born with leaders used to waterfall.

I agree with almost everything in this article and I completely agree with what Tootie wrote above: "The fundamental problem that drives most agile failures isn't in the team's execution, it's in the business' expectations. One side is signed up for incremental delivery, and one side is set up for a fixed scope and deadline and the result is misery."

As the article rightly noticed being Agile is not just building faster. It is delivering "value faster". That also means: "do not build what does not need to be built" and every 2 weeks be willing to scratch what you have worked on. Instead in most of the big companies the business people present shiny objects to the leadership and their goals is to build those shiny objects. Nobody wants to say: "We were wrong". The result is a few projects were people worked for 1 year and then the project is abandon. Even if the idea was good, nobody wants to work on little improvements.

It seems that Agile, Scrum, and most of classic agile lingo have a bad connotation for most of the people in these days. Some (over) simplifications that were made to make Agile easier to adopt ended up making things worse.

People in big corporations are used to the "flavor of the month" Most of the people really do not look for a revolutions there. They are just looking to how they can seem smart to impress their bosses and being promoted. That how Scrum got in. And that is why it will never realize its full potential there. They do not want to change how they think and how they do things.

I run a pretty successful team inside a bigger organization. I wanted to use story points. Start from scratch and have the team get a sense for what 1 point is. And my boss really wanted us to use story points. He only had one requirement we had to use this equation 1 story point = 1/2 day of work...

Trying to explain him that one of the reason you use story points in the first place is to stay away from time did not go anywhere. All the other teams were using that equation and we could not be the only team that was not to using it.

Frankly speaking, I think it is hard to work at the "inter-team" level if you cannot use the same unit. That's why in Agile you want truly autonomous teams. But big organization were not organized in that way. So you need to re-organize the company in a different way... but assuming there is the desire, who is going to drive that change? Who is going to take the risk of changing something that work in some way in a way that is not going to work too well for a few months?

In my humble opinion, it is almost impossible to change an existing a waterfall organization and make Agile work inside of it. Only a leader with a lot of trust by everybody in the company and a lot of courage may be able to make it work.

Most of the people that hate Scrum, Kanban or other flavor of Agile had experience with a "Cargo Cult" Agile preached in big companies and usually end up thinking that Agile is sooo screwed up. A big corporation will usually embrace embracing some things that usually really help (ex: standup meetings) and change a ton of other things that will make things worse.

You like the Agile mindset, you are going to be a lot more successful if you start from scratch with new young people or people that really want to commit to make things great.

For a new organization is easy to start with a simple "Agile template" and make sure that the team is really onboard with a continuous improvement mindset. Keeping regular retrospectives will make things evolve quickly in the right way.