Hacker News Reader

Comment by redcodenl | Hacker News Reader

Regarding Conway's Law:

We have found that by changing our software/system architecture we have also inadvertently changed our organisation structure.

- Inverse Conway Law or just Roy's Law ;-)

Before we had four cross functional teams, working on a single application, everyone felt responsible for it, worked overtime to fix bugs etc, we had good communication between the teams.

But after we switched to microservices the teams became responsible for just a part of the system, their microservice(s). Whenever we had an outage, one team was left to fix it, the others just went home. They stopped talking to each other because they didn't share any code, no issues... they stopped having lunch together, some things got way worse in the organisation, all sparked by a 'simple' architectural change, moving to microservices.

View on HN

That reminds me of a place I used to work at, where initially we had DBAs embedded in the teams. They switched that and the DBAs were all grouped together and all hell broke loose. They were always have meetings, throwing out emails about they were dictating this and that, and had very little direct communications with the teams they were supposed to be supporting.

I ended up leaving during the peak of all of this, but in an exit-interview, a director asked me about the problems this was causing.

> They stopped talking to each other because they didn't share any code, no issues... they stopped having lunch together

Was that accompanied by any growth in company size? I've found that this happens when a group grows past about 15 people even if the structure doesn't change.

This outcome could be considered a feature of microservices: by abstracting the functionality into more tightly-contained units, failures are more isolated.

Sounds like the organization needs to do other things to keep people from getting siloed, though that gets increasingly difficult at scale. Well-defined SLAs (along with monitoring and reporting of those SLAs) are also necessary so that microservice failures can be understood in the right context.

This YC talk from Amazon's CTO on how they grew to a microservice model and team structure was really interesting: https://www.youtube.com/watch?v=adtuntQ8rh4

I have seen new systems/software implemented just to isolate/remove parts of an organization. Worse I have seen it done when the existing system/software was just fine.

> They stopped talking to each other because they didn't share any code, no issues... they stopped having lunch together, some things got way worse in the organisation, all sparked by a 'simple' architectural change, moving to microservices.

Honestly, this sounds like an improvement.

How is people not talking to each other an improvement?

Well, from a systems standpoint, it means a given "problem" is isolated to a single service, and is therefore not impacting the other services or interrupting the work of the other teams.

But culturally, it would be nice if people helped each other out from time to time...

That's only true if any given problem is isolated on a single service, because while you are making intra-service problems much cheaper and faster to fix, you are also making inter-service problems almost impossible to fix.

Good point. I've seen situations where nobody would take ownership of a bug and fix it - you just had teams pointing fingers at each other...

This is precisely a symptom of unclear ownership responsibilities.

No, it can be an issue of it being unclear in which component the bug actually is.

Which is a clear sign that your components are not sufficiently separate.

Um, no. The real world is not that simple. It would be nice if it were.

Or perhaps your statement is correct, but in the real world components are never sufficiently separate. So, while your statement may be correct by definition, it is not useful.

I think you really nailed it with this one.

I can imagine there being a normal distribution of 'separateness' of software and the rare top tail-end of the distribution gets it perfectly right, most are in the middle somewhere between 'service oriented architecture' and 'ball of mud' and some are just at plain ball of mud.

You are misreading my words.

Perhaps so. Would you care to explain? Your statement here doesn't give me much to go on.

Well I meant it quite literally: if it is not clear which component has the bug, the components are not separate enough. You also say:

> in the real world components are never sufficiently separate

But the separation of components is not an issue of "real world", it is a function of design and implementation. It is absolutely to the developers how independently the components will be implemented; if there is no way to test them in isolation, then they are not really separate components.

Take this website, and your browser, as an example. They are obviously connected, as you're using the latter to access the former, but they are completely independent: you can access other pages with the browser, and you can use other methods (other browsers, or curl, or Postman etc) to access this page. Each can be tested separately, and even when they are used in conjunction they don't directly depend on each other, but rather on a set of standards (HTTP, HTML, CSS etc) understood by both.

Yes, but then you find web pages that attempt to determine which kind of browser requested the page, and changes what is sent back in order to work around the broken-ness of specific browsers. Yes, it's supposed to be a nice clean interface specified by standards. But in the real world, as they say, all abstractions leak. "All" may be an overstatement, but the problem is real. We never separate things cleanly enough.

This statement is true, depending on how you define "enough".

Sure. Look, despite the impression I may have given, I'm not arguing for separating things badly. The cleaner the separation, the better. It really makes a difference.

Dunbar's number. As humans, we only have room for so many relationships.

Oh, it is definitely a problem from a social/cultural standpoint. But from the point of software architecture (and, therefore, development organisation), too much (or even any) communication between teams working on discreet, separate units can become detrimental.

It is perfectly fine that the people communicate, and even helping each other to improve tech skill should be encouraged; however, decisions about their respective products should be contained within each team, with clearly defined interfaces and usage documentation.

> decisions about their respective products should be contained within each team, with clearly defined interfaces and usage documentation.

In order to make those decisions and define the interfaces, you need to know a lot about how your software is going to be used. That will be much easier if you have good communication with the other teams and understand their goals and motivations.

I disagree. In my experience, direct coordination on interfaces tends to create unnecessary special cases (hey, can you add this field to your API, just for us?) which add complexity and make maintenance more difficult down the line.

The main advantage of distributed system, and particularly microservices, is the ability to have each system completely independent: individual components can be written in different languages, running on different platforms and use completely independent internal components. Basically, it is just like using an external library, component or service: the authors provide documentation and interfaces, and you should be able to expect it to behave as advertised.

> In my experience, direct coordination on interfaces tends to create unnecessary special cases (hey, can you add this field to your API, just for us?) which add complexity and make maintenance more difficult down the line.

If you just implement all requests directly, you're for sure going to end up with a horrible interface. You should approach API design the same way that UX/PM approaches UI and feature design: take the time to understand _why_ your partner teams/engineers are requesting certain changes and figure out the right interface to address their problems.

Oh absolutely, but direct communication between teams is not the right method for that. Which is why every product, no matter how "micro" a service, needs to have a dedicated product owner/manager who is responsible for defining functional requirements.

Edit: I just noticed that "PM" in parent comment. Basically, product managers are not just for UI and customer-facing products.

The idea that "functional requirements" can be decided independently from "technical architecture", as opposed to in interplay with each other, is exactly the opposite of what I've learned from good experiences with "agility", although some people seem to somehow take it that the opposite.

But yes, you can never just do what "the users" ask for. The best way to understand what they need is to be in conversation with them. Silo'ing everyone up isn't going to make the overall product -- which has to be composed of all these sub-components working well with each other -- any better.

> The idea that "functional requirements" can be decided independently from "technical architecture"

Oh it absolutely can; it's just that it usually is not a good idea. But I'm not talking about the process of reaching that decision, I'm talking about the responsibility to reach them. Functional and technical decisions are separate, but in most cases should be defined in conjunction.

> Silo'ing everyone up isn't going to make the overall product

This is true for certain types of product, and less so for others; you need to clearly understand what type of product you're building, and be ready to adapt as it changes (or you have a better understanding of it). In a nutshell, the more compartmentalised a product is, the isolation between the teams becomes more beneficial. Which brings us full circle back to Conway's Law.

> But I'm not talking about the process of reaching that decision, I'm talking about the responsibility to reach them

Of course you're talking about the process. Your claims that "direct communication between teams is not the right method for that," and "communication between teams working on discreet, separate units can become detrimental," for instance, are about process.

I don't think this conversation is going anywhere, but from my experience, lack of communication between teams working on discrete, separate units (that are expected to be composed to provide business value), can become detrimental. And that's about process.

Presumably if you don't like other people then it's an improvement. At a guess I'd say that covers about quarter of our industry.

Probably we can agree that a company will be more productive if the engineers are learning from each other and generating ideas together? So if you've got a team of engineers who don't like working with people, it's probably in the company's best interest to set up a structure that explicitly encourages more communication.

This depends very much on the team size. Having a team of 20 people communicate in the way you describe below is insane overkill. Having a team of 100 do so may save everyones' sanity.

Moving to microservices is anything but a simple change. Your experience is one example of why microservices are not automatically a good idea. Normally, the advice is that they might be a good fit if you have isolated teams to begin with, and for different reasons.

it's almost like Bezos intentionally issued the microservices mandate at Amazon to discourage Amazon's engineers from unionizing.

> “Before we had four cross functional teams, working on a single application, everyone felt responsible for it, worked overtime to fix bugs etc, we had good communication between the teams.”

This actually sounds very dysfunctional, but with the type of positive PR spin that product / executive management wants, basically anyone who believes “cross-functional” is anything more than a buzzword.

Would love to know what the engineers thought about working in that environment (which sounds like a monolithic, too-many-cooks, zero specialization situation likely negatively affecting career growth & routine skill building).

Aside from the working overtime part, what sounds dysfunctional?

Types of bugs or failures are not separated into specialized areas, rather it’s one “cross-functional” unit. It’s like building a monolith class in software instead of following basic principles like Single Responsibility Principle and organizing workflows according to independent specializations.

This part is often much worse than the overtime part, because it means you’re expected to sublimate your personal career goals in favor of whatever arbitrary thing needs done for the sake of the cross-functional unit.

When I hear someone describe communication between cross-functional team members as “good” or “effective,” then I know it’s a big lie, and most probably it’s a disaster of overcommunication where product managers or non-tech leadership have a stranglehold on decision making when really engineering should be autonomous in these cases exactly according to independent specialization.

Could less overtimes be considered a positive outcome?

Haha, this is so similar to what happened where I work in SF that I feel you’re a coworker of mine.

Reply on news.ycombinator.com