Comment by solatic | Hacker News Reader

Comment by solatic | Hacker News Reader

Parent Full thread

At its core, the true underlying problem is that you can't actually fully control another person's output while still holding onto any genuine agility. The design is going to leave out some Product features, the code is going to diverge slightly from the design, QA will find some things the code didn't account for, DevOps, Security, and Legal will each chime in with other things that weren't considered.

If the organization requires that everyone is happy, then everyone gets to review the work at every stage (in order to express unhappiness and get it fixed before proceeding). The inevitable result is a Waterfall process to gatekeep how work moves to each stage. The inevitable result is a lack of agility.

But if the organization can lean into the Truth that Nothing Is Perfect, that your product is not perfect, your design is not perfect, your code is not perfect, your scaling, observability, security, and compliance are not perfect, then suddenly, nobody needs anybody's approval. You ship, and you ship quickly. Then you focus on learning and iterative improvement. Make things better over time, but not necessarily right now.

Getting rid of Waterfall is learning to let go and to live with imperfection.

The problem is that some things must be perfect, because they expose the company to significant risk. You can iterate on a feature but not on compliance.

You can iterate on compliance _implementation_ if you keep some tollgate that guarantees all compliance business requirements are eventually met. In my opinion this is not the same as waterfall, and leaves room for experiments and iterative learning.

The (very) tricky part, in my experience, is to have these requirements captured in implementation-agnostic way. Even strict accounting reqs require huge amount of work for someone to deconstruct them down to actual compliance requirement, instead of something solution-specific like "we need to store X in component Y".

Not everyone wants to invest into this just for agility sake.

> eventually met

That might work in some fields, but consider life sciences and pharma where some unmet functional/compliance issue could be injurious to the company, a patient, or both.

Waterfall-ish workflows are hard to avoid in such cases. We learned to simply do iterations of waterfall and speed up the validation via automation as much as possible.

Even if that's true - I'm not sure it is, particularly in very early R&D phases with no external impacts - all that means is that non-waterfall doesn't work absolutely 100% of the time, which nobody is saying it does.

Any time an engineering methodology is discussed, people come out of the woodwork with their specific example of one time where that methodology won't work.

I worked in the pharma space building document management systems that supported clinical and pre-clinical processes (mostly clinical, some R&D).

After a while, one starts to understand the need for the rigor. If some cluster of patients suffers some injurious adverse effect pre- or post-market, the FDA and a bunch of lawyers wants to trace the history of development to understand how to assign responsibility. Software systems in this space are validated quite rigorously against their functional and technical design specifications to verify that the records produced are traceable.

Software validation in this space works kind of like GPL licenses: the sponsor (like a Merck or a Pfizer) is responsible for validating software from vendors when it is supporting a GxP process. Those vendors are then responsible for validating not only their systems, but all dependent systems as well. (A lot of it is risk-based and obviously a lot of the methodology, guidance, and interpretation has evolved since the cloud era. But we used to have to take our customers on tours of our data center and their auditors would match the specifications of the hardware!)

I think it's one of the reasons why -- despite this process being extremely high in "schlep" -- there are few startups in this space because the barrier to entry is quite high. Even having 15 years of domain experience and running a team delivering production systems used by pharmas to run their trials, I don't think I'd want to do a startup in this space.

There's a really great HN thread on this that resonated with my experience that I bookmarked a while back: https://news.ycombinator.com/item?id=33545209

In my experience at pharma there has been an enormous divide between R&D and production for software development. Precisely for the compliance implications. In cases where it hasn't been a perfect black/white differentiation it was always dirty laundry where some subset of people knew they were sitting in a grey area and hoping that compliance officers/auditors wouldn't notice.

The PCI Compliance audit commission would surely have some thoughts about this philosophy.

You can just section off the bits that need to be PCI compliant.

It would be a good architectural decision, actually.

You can iterate on compliance too. Just ask Uber.

... and leave ops/ devops / are team to pick up the pieces (or put the pieces together) because now your too busy to iterate because your shipping another new thing?

1. If you're agile, you're not too busy to iterate because you're shipping another new thing. You're able to come back and iterate if that's the most valuable thing you can work on.

2. When waterfall delivers the wrong thing because the requirements from two years ago weren't updated to reflect two years of new reality, someone has to pick up the pieces from that, too. It may not be devops - it might be sales instead. But it's still a problem.

The standard fix for this is to put the devs on call.

Recurring operational issues get fixed mighty quickly when J. Dev has to wake up a few times at 02:00 on Saturday.

Heck, design and architectural issues suddenly get a lot of scrutiny at the whiteboard phase and people decide they don't really need Kafka or Kubernetes or Mesos or GenAI anymore.

If anyone is getting 2am calls more than extremely rarely, the process is already seriously broken.

That thinking is one reason why I left engineering and I don’t see myself coming back to it anytime soon. I don’t like being on-call, I’m not qualified to do Ops-like decisions and I haven’t seen this as the practice in most other industries - Boeing doesn’t have staff flying planes for the airlines, for one.

Compliance is a set of stories/tickets/requirements like any other, with a priority to be assigned and eventually worked and reworked at some point in the process. There’s nothing wrong with not addressing it as the first thing - it just blocks release. With that, it will eventually get worked on, hopefully at the time where the pieces that need it are understood and the work to reach it is understood.

At a $LASTJOB all the devs had to be on call for some services and literally nothing improved. Still haphazardly throwing code into production, poor monitoring tools, and no time or desire to make stuff better. On-call was just seen as shitwork and devs would find other teams that didn't have on-call to transfer to.

That's kinda how it works at my current job, where a support rota is manned by members of the various development teams (front end dev, backend dev, app dev, platform ops, etc). You can indeed get woken up at 2am on a Saturday, and expected to fix whatever mission critical issue has popped up at that point.

It doesn't seem to have changed much as far as the architecture process goes though. With a few exceptions, the things that pop up for support to handle are usually either 'something broke on the content side, and the editors can't fix it', or a third party broke (glares at Google and Tag Manager, Musk and Twitter/X, etc)

This is why I am a fan of build & run teams instead of having a separate ops org which is responsible for keeping things running. The incentives aren’t aligned and almost always result in silos and excessive outages. The shared responsibility means no one is actually accountable and solvable issues persist for years. Almost every client I work with who has major ops issues have separate teams handling build and run aspects of their projects.

In 15 years of experience here and there, only one has had on-call devs. I’ve come to the conclusion that neither dev nor their management care if someone else’s phone rings at 02:00. I’ve also seen open hostility to 1) the very idea of dev going on-call, and 2) technical recommendations for solutions to recurring issues. So I think it’s not quite “the standard fix”, in fact I think it’s pretty unrealistic.

> Getting rid of Waterfall is learning to let go and to live with imperfection.

To a degree. But I'm pretty sure one of the reasons why software quality is so much lower these days is because many teams take the idea too far.

I think this is totally right. But some things really do need to be as close to perfect as you can get them. The things in that category that I've run into most are security, privacy, and legal compliance. But in other applications, safety issues would also go in this bucket.

So those things really do need a proactive gate, or maybe in some cases, at least an extremely fast reactive response capability.

But this just means it is more important to identify the smallest set of considerations like this, in order to give them special treatment, rather than requiring higher scrutiny for everything - if everything is special nothing is special - and to invest in ways for work to be done without implicating those special considerations.

Reply on news.ycombinator.com