159 karma · joined February 9, 2010
Software Development Articles: https://github.com/aaronjensen/software-development/#software-development
Here’s a quick guide:
Working on a feature branch with someone else and you have local commits and they have remote commits? Rebase
Need to rewrite history? Rebase
For just about everything else, there’s merge.
Example:
EntityStore build method records the specifier argument when given
Article about it: https://github.com/aaronjensen/software-development/blob/mas...
It doesn’t have to be like this. This is a choice. There’s no reason a team cannot work towards a world where they can push a small fix to master directly after running their 5 second test suite on their local machine.
We are used to squalor, but it’s not necessary. It’s the result of a series of choices. There are other ways to do our work that maintain continuity of productivity. Old projects can feel like new projects.
Any time someone writes something like this, they are normalizing slights against our fellow developers. They may not recognize it as such because it’s all they’ve ever known, but we should (collectively) know that better is possible and strive for it.
By the way, the entire premise that you should organize your software based on the way teams are organized is the flawed thing. Natural designs emerge when designing software. If you must have multiple teams, it may make sense to distribute some of those partitions that emerge across those teams. Or it may not. What absolutely does not make sense is creating partitions to account for your multiple teams. That's design-by-making-stuff-up and it's as doomed to failure as the REST-based-microservices fad.
I got it from here:
> This was specific to the tendencies in other replies to defer to your (or your team's) brilliance/experience ...
and
> Being so cocksure and backing it up with statements that you need to be as brilliant as you to "understand" is such a laughingly large red flag...
Given that at no point did I point to our "brilliance", I assumed you refererring to my pointing to tacit knowledge:
> It's honestly a hard question to answer because the real answer requires tacit knowledge...
> ... because much of what we do requires tacit knowledge to see the benefit of ...
So if I'm mistaken, I apologize, but please do point out where I claimed brilliance. I am certainly claiming experience, and expertise, but those are earned, as they are in any profession. It's also true that no one in this conversation (including you) other than me have actually seen our code, and therefore would be ill-equipped to judge it. That is, they do not have the experience of our code base. Rushes to judge it based on pre-conceived notions only reflect a lack of dilligence, integrity, and/or awareness.
> the resulting quality of the output however can usually be recognized by everyone...
Have you seen our code? Or have you seen a few mentions of some of the things that we do and you are using that to fabricate an idea of what our code is?
> So therefore I'm a bit suspicious of anyone that claims that their idiosyncratic setup is actually simple
You are of course free to be suspicious. A significant part of the experience necessary to judge it would be to actually see it. We are, at the end of the day, dancing about architecture. You cannot see our code, you cannot see our actual set up, which typically would mean that one would have essentially nothing to say about it. If you'd like to refute particular practices I mention, that's fine. But please, keep your presumptions to yourself, or at least ask them in the form of a question (e.g., "artificial partitioning").
> ...would be through a pile of design patterns inspired abstractions that now everyone needs to be able to juggle at all times.
We have relatively junior developers on our team. There are a small handful of common patterns that are used repeatedly throughout our code base. We strive to eliminate special (unnecessary) variatation so there are always exemplars and/or documented norms. We don't have 10 varieties of "service objects". We don't have callback hell. They do not struggle with these things. They struggle with other things, as they are relatively junior, but we support them. Again, frankly, you have absolutely no idea what you are talking about and you are continuing to attempt to make up for your ignorance (of our code base and what I am saying) with hubris and presumption.
I just saw this edit from you:
> That said, if a code base keeps requiring you to make the correct design decision using subtle knowledge, that's a fragile situation and something seems off.
You and I have two different ideas of expertise and software design and they are irreconciable. Maybe in 15 years you can look back on this conversation and recognize that there was always land beyond the horizon. Or maybe not. You are the doctor that is refusing to wash his hands before surgery because you still believe in bad humors. There are things you do not know, and you are stubbornly refusing to recognize them. You have probably worked on some pretty horrific code bases, and you can probably back up every one of the things you are projecting onto me with personal experience, but you have drawn the wrong ultimate conclusion. You have drawn the conclusion that software design does not exist and does not have consequences and that anyone who claims to do it is a charlatan. That may even be true much of the time (goodness knows that's what I'm saying about many people who claim to be doing software design). Judge me as harshly as you wish for making this assessment. Thankfully, we do not work together and we will be unlikely to cross paths again in the future.
I touched on this in my reply to the other user, but at this point, I believe you may just not know what tacit knowledge is. You continue to think that I am pointing to our own brilliance. I'm not. I'm calling a spade a spade. I recognize what it takes to gain particular types of knowledge (tacit, or subtle knowledge), and I recognize that it's this reality that prevents most conversations about techniques from being fruitful.
Each participant will put their own experience behind the meaning of their words (and worse, their conversation partner's words) and it will prevent them from recognizing what one another are saying. The only way to have a fruitful discussion is for both sides to be capable of recognizing when that is happening. Since, in my experience, most people aren't -- they'd rather die on their hill than recognize that the person they are talking to is simply on the same hill but sees it differently, or they are on a different hill that really is better, but it cannot be seen as such yet, because it is over the horizon of their knowledge.
I don't actually know if you are interested in understanding this more or if you joined the conversation just to try to put me in my place, but if you are, here are some articles that may help:
https://madabout.software/articles/subtle-knowledge-crude-kn...
https://madabout.software/articles/design-is-subtle-knowledg...
Once you can recognize that there is subtle knowledge you might actually see my pointing to it as attempting to keep the conversation from devolving into exactly the type of thing that it tends to devolve into. Or not.
With all due respect, if you think you are talking about anything that I was attempting to describe, you are not. Everything that we do is based on simplicity and building the most simple things that we can. We may have different definitions of simplicity however. For mine, check out Rich Hickey's "Simple Made Easy" talk if you haven't seen it.
We simplify things by de-complecting or, avoiding entanglement. Having separate repositories is just one way to re-enforce the simplification that we put in place with design. We call that structural design (because it's the design of the actual structure of the application).
> on that note, it sounds like hmeh is trying to work around this problem with abstractions by doing a significant amount artificial partitioning.
I can guarantee you that none of the partitioning is "artificial". I think at this point that you have projected so much of your experience into what I am saying that it's impossible for you to see anything I say in any other light. This, by the way, is what I mean by tacit knowledge. I say X, and you either hear Q, or you hear XIENRSTIENRSTIERNST, either way, you cannot possible understand what I am saying. This isn't a negative reflection on you, or a positive reflection on me. It's simply the way that tacit knowledge works, and it's why most "internet arguments" are wastes of time.
> People promoting these things will of course then just blame it on the dev and call them average, or lacking in experience, because they can't identify and combine the various clever patterns correctly, they "are doing it wrong", and that's where I believe the above narcissistic tendencies I called out comes from.
And yet, it is literally lack of experience (and ability) that prevents people from doing things they cannot yet do. That's how expertise works. Pointing that out is not narcissism. A gardener does not have the experience (or training) necessary to do brain surgery. This isn't a mark against the gardener. The difference in our industry is that it is filled with expert beginners. People who think that because they make 6-figure salaries after their limited education and experience they can now understand everything there is to understand about software design. This just isn't the case. If a person sees that as a slight to a fellow human, that says more about them than it does about the person pointing to that reality.
Yes! Absolutely. With one key difference: You can't commit to multiple repos at the same time. You can certainly have them all checked out locally and search across them all, and we provide scripts to do that. Example from the Eventide project: https://github.com/eventide-project/contributor-assets
This is an open source project. Our project is based on many of the ideas and techniques. They've held up for 7 years or so.
How so? Why don't we put our Rails applications in the Rails repository?
> And IMO invoking Plato to make a point is not helpful. Argumentum ad antiquitatem
What I am pointing to is a common thread. Plato, Christopher Alexander, Edwards Deming, etc. What I am arguing for is anything but common tradition. It's counter to what is current "common knowledge".
> This goes for the “30 years” comment as well: it’s entirely conceivable that they’ve been doing things inefficiently for 30 years.
Yes, entirely possible. And if we find a better way to do things, we will be first in line.
> More likely that there is such a huge sunk cost that they are heavily invested in justifying it.
What's your evidence of this?
Yes, let's. I have no problem with push back.
> You have experienced what happens when web developers cosplay as software architects.
I agree that this is overly snippy to the point of being counter-productive. It represents a particular emotional frustration with the state of our industry. We knew what ended up being microservices (web API servers making web API calls to other web API servers) was a failure mode. We knew you couldn't just introduce network hops and call it "architecture". So yes, I'm annoyed about that such that I'm willing to call out nameless "web developers", but I understand how that can create discomfort.
> That is, you experienced an average team doing average work misguided by average bad advice. It says absolutely nothing about software design. It only speaks to the fat part of the bell curve doing what it always does.
Indeed, and the person I was responding to effectively acknowledged this. The sooner we can recognize that the skill curve/technology adoption curve are real things, and crossing the chasm takes hard work, the sooner we can stop leaving people behind said chasm. It's not easy. Nothing I said here is untrue. If it is, please point it out.
> because, unlike them, I have significant material experience in both camps.
Again, I'm pointing to a material difference. Most people who are attempting to refute what I am saying have never done what I am saying. They are doing it from a position of fear, uncertainty, and doubt, or worse. I've been where they are. I've fought the fight they are fighting and, thankfully, I lost, and was introduced to new ways of doing things and seeing things. If a person has done what I'm discussing, I would expect them to say that and tell me why it failed. Instead I get people telling me it can't work and I'm incompetent, etc.
> It’s hard to imagine worse advice. > Just read the author’s bio. This is a person that appears to have zero software design experience writing an article telling you to ignore software design and just respect your team configuration. I call this Conway’s Confusion.
Putting these together because they are about the OP article and the OP. "hard to imagine worse advice" is hyperbole, but it is bad advice. You can't force fit concepts to teams. I mean, you can try, but you'll always end up with unnaturalness.
The actual OP's bio talks about their career. None of it mentions software design. If a person with zero surgery experience started writing about how to properly set up operating rooms, you can darn well bet they will be called a charlatan and called to the carpet. If they aren't people may die. Software isn't that serious usually, but it's not hard to imagine that there have been billions upon billions of dollars flushed down the toilet for the sake of poor software design that no one speaks out against.
> But please, by all means, continue to spread disinformation and keep us in the dark ages.
Yes, overly cynical and unnecessary. I made my point prior to this and I didn't need to add this.
> That sets a good friendly tone huh? If you can't take push back, don't be an arse to begin with.
Some was unnecessary yes, thanks for calling it out. The rest represents what I think is healthy push-back against an orthodoxy that causes more harm than good. We can achieve more as an industry and make our way out of the realm of hobbyists and into something more serious. Many of us call ourselves engineers, but nothing we do resembles anything that people who are actually classically trained engineers do.
> > And which part, specifically sounds incompetent? I’m happy to discuss that.
> Your arrogance precludes a fruitful discussion. But I believe that this needs to be called out, if nothing else to nudge other people to also do it when they see it, or to, albeit much less likely, nudge you towards eating some humble pie.
Just to be clear: you said that what I was saying sounded incompetent. I asked you about that, and you're telling me I appear to be too arrogant for you to tell me why I sound incompetent? Please, tell me what I said that sounds incompetent. Let's move past the ad hominem portion of the discussion now and talk about the actual substance. I'll do my best to refrain from hyperbole and unnecessary snark.
Where did I say that, exactly? I’ve said nothing about intellect. I’ve mentioned tacit knowledge multiple times, because I have enough experience to recognize how ludicrous a lot of this sounds. It sounded that way to me at first until I experienced it and saw the pieces fit together. That’s how tacit knowledge works, especially when it is counter to average knowledge. It’s the same reason American industry smashed Japanese cars in demonstration instead of listening to them when they said large batches were bad (I’m simplifying here somewhat)
And which part, specifically sounds incompetent? I’m happy to discuss that.
By the way, it's decidedly difficult to challenge a common belief, dare I say, orthodoxy, without sounding insane, confident, or even cocky. Could I be better at it? Of course, and I work at it, but I also am not here to make friends. I'm here to provide a perspective that I believe is sorely lacking from the development community and is drowned out by the orthodoxy, its acolytes, and its beneficiaries.
> I really hope your colleagues are onboard with this and that this isn't your own dogmatic crusade dragging them along.
Given their tenure on the team and their other options, it’s a pretty safe bet. It’s not my crusade, btw, it’s our teams goal to manage a relatively complex project with a relatively small team.
Also, someone has to say it, so I will. You have now dropped this conversation, which was about the work, into ad hominem and straw man attacks. That’s one of the worst part about this industry: people’s inability to debate and discuss without attacking people or glorifying celebrities.
But, why don't I go ahead and give you a real quote.
Of our 400+ repos, 100 or so of them are deploy projects. The only time they need to change in batch is if one of our boilerplate scripts changes. This is infrequent, but when it does happen, we have a "project-scripts" repository that contains our boilerplate scripts as well as scripts to update them across all of our repositories. This doesn't always require individual verification, but when it does, it's typically a matter of installing dependencies and starting the application.
Many text editors are capable of doing "single search and replace" across multiple repositories. All of our repos are in a single project directory, so we have a root, just like anyone with a monorepo does. Rarely does that "single search and replace" touch more than a small handful of projects. So, except in the extreme case, you are only running a fraction of the tests or verifications when you make one of these "search and replace" changes. Compare this to a monorepo, where, unless you have specialized tooling to slice your test suite while considering your package dependency tree, you end up needing to run all of your tests.
If you are responsible and your project is in control, you run the tests on your laptop first, and then (typically) you push your commit and it runs in CI. Actually, more typically, you push your commit to a branch, open a pull request, have it run in CI, have it reviewed, then merge it, where it runs in CI again. Each time it ran in CI, it ran your entire test suite (usually).
We don't do most of that. Most of what every team takes for granted, we just don't do. So, when someone imagines having to do their typical process for getting a change integrated 100 times, they are right to fear that. But, they are creating a false equivalence. We don't do that, because we've eliminated all of that waste from our process. What we do 100 times (again, the once or twice a month that we do it), it takes a fraction of the offort what most teams do once to get a single change in.
There's a reason that Toyota ate the west's lunch when it comes to manufacturing. They understand just-in-time (one-piece flow) and small batch sizes. They know how to address problems at their root and systematically eliminate waste.
Most in the software industry know how to do one thing: apply the thing the latest dev celebrity said in their latest tweet or blog post.
Here's the difference between me and that celebrity: I'll tell you straight up not to do what we do. I'll tell you straight up it will take you years with guidance to get anywhere close to what we do. I'm just telling you it's possible and we don't actually need to live in software shanty towns our whole lives.
But, I would challenge you to challenge anything that you saw as satire and try and wrap your head around how a sane, rational person could, with a straight face say "that's not a problem" and be absolutely right. Feel free to call something out specifically to me, and I will back it up. I'll warn you that you may still not agree with me, because much of what we do requires tacit knowledge to see the benefit of. And once you do, it's self-evident. A person must be studied in Lean and design principles and have seen the trajectory of several software projects over the course of many years.
See if you can understand what are the actual problems in software. What causes projects to fail and teams to have to hire 100s of developers to maintain a semblance of the productivity they had for their first year or two of development.
Try to record your setbacks in a daily work log (we all do this) and address root causes. Do this for 10-20 years and see where you end up.
Also, no one said "microrepos". Don't put the current fad's fixation on what I am describing. I'm simply describing structural design. Does thing X and thing Y need common behavior? Is it actually generalized behavior? That's what we have called a "library" since we've been building non-trivial applications out of software.
Are there two disparate business processes that don't need to share states, and you know they don't need to share states because you've done the design work? Those are two separate components. Pub/Sub exists for collaboration between those components as necessary.
I'm not trying to say that our current technique for all things is the end-all-be-all. That would deny the continuous improvement mindset (i.e., the relentless pursuit of perfection) that got us where we are. But, there are things that won't change. Smaller batches will always be (generally) easier to manage than larger. Taking something that has high afferent coupling and adding specializations to it will always be a mistake (See Fat Model, Skinny Controller).
The reality is that there are software design mistakes that are knowably mistakes. They've been known for some time as well and can be measured. Those are the things that we avoid and those are the principles that have guided where we are.
I know this sounds highfalutin, but there is someone on our team that has been doing this for over 30 years (by "this", I mean continuous improvement/seeking perfection) and has studied under many of the greats in our industry. We aren't just making this stuff up.
> 10 of those repos were owned by someone who left the company a year ago
That's not how it works. Repos aren't "owned" by someone. Each repo stands on its own, is clearly testable on its own. Every one of our teammates is capable of updating every one of our projects because there is consistency between them.
If upgrading Rails or any gem scares you, you have other problems to deal with. For me personally (I'm relatively fast on this team) I can update Rails in all 20 of our Rails applications in maybe 30 minutes unless Rails caused a setback (this is somewhat frequent between propshaft and turbo). If Rails caused a setback, we have two dependencies (rails-application-operational and rails-application-development) that we can either include a patch or pin Rails. All told, it's a quick process.
You know what we don't have to spend time doing? Tracing callbacks across 30 entangled ActiveRecord models. Or anything else that comes from entanglement. We also don't need to entertain palliatives like packwerk and the like.
Also, I never said 400 components -- we aren't there, yet. It's ~100 components. It's just 400 repositories. 100 of those are deploy projects (they simply contain scripts, a dockerfile, helm values, etc.) most of which is boilerplate.
By the way, I know how it sounds. And I know why a person would fear it. And, I know why they’d be incorrect in their assessment, because, unlike them, I have significant material experience in both camps.
> More directly, what's wrong with a monolith at the dozen person team size?
Great question. It's honestly a hard question to answer because the real answer requires tacit knowledge, but I'll try anyway because you asked. I'll link a couple articles of longer form writing that may help to answer as well.
The benefit ultimately is productivity. That productivity comes from autonomy, which comes from partitioning.
Each component can be worked, tested, and deployed on its own. We never need to spin up more than one at a time, which means any time we are working on our code, we are typically working on 1/400th of it, and we know we are because we have 400 separate GitHub repositories. We can't accidentally get peanut butter in our mayonaise. Each part is independently built and tested.
When it's time to upgrade Rails, we can upgrade one application. This is 1/20th of our Rails code. That's a smaller batch. We know from basic flow principles that (see: Principles of Product Development Flow, Lean, etc.) that small batches are better for productivity.
We rarely have merge conflicts. We don't do pull requests. We branch when we need to. We are often working on a brand new, never been deployed project. We don't have our tests running in CI. Our tests run in less than a second in most projects on our laptops. With those that have UI, our tests typically run in less than 30 seconds which include thorough Capybara (UI Interaction) testing.
When we are working in any application, we only need to consider its direct efferents when making a change to any interfaces it exposes. We can trace those changes through easily when we need to make them. Because we practice software design, we don't often need to make large sweeping changes, but when we do, we can do them methodically without disruption.
The "worst" part is that some things are tedious. But we can automate tedious things. We can invent around tedious things. We used to have to deploy 15 web applications any time we had a style change. Now we deploy one application that supplies the CSS for all our applications via SSI (server-side-include). If we need to deploy 70 back end components, we can script that (with manual verification at each step). We can even spread that out across a couple team members and do it as an ensemble. It takes about an hour to deploy everything. We maybe do this once a month or so (when something highly afferent changes -- and guess what, we structure our design so that afferent things change as infrequently as possible).
Most things we work on we work until they are "complete". That is, they rarely change after that. We have components that have been in production for 3 years and haven't been substantially touched aside from Ruby upgrades and the like. They are still perfectly in control.
The list goes on and on.
Here are a few articles, feel free to poke around the others and ask any follow up questions.
Partitions and Compositions: https://github.com/aaronjensen/software-development/blob/mas...
The Mythical Monolith: https://github.com/aaronjensen/software-development/blob/mas...
A dozen or so is the entire software development organization.
Edit: Indeed! If you have any other questions, ask away. Most people think what we do isn't actually possible, but it is.
You have experienced what happens when web developers cosplay as software architects. That's when you get "microservices". It's a travesty, and it has ruined an entire generation of software developers who now believe that partitioning and autonomy are just myths.
So, as a blunt statement - your experience is typical. That is, it's average. That is, you experienced an average team doing average work misguided by average bad advice. It says absolutely nothing about software design. It only speaks to the fat part of the bell curve doing what it always does.