In Defence of Monoliths
techblog.bozho.net
techblog.bozho.net
An important meta-idea that encapsulates all of this is remember to solve your business problems as your first priority, and implement technology in service to that goal first and foremost.
It's far too easy to put various technology decisions in the driver seat, ahead of the business problems. This leads to all kinds of anti-patterns over the years. For example, I saw software development organizations destroy themselves trying to do 100% pure UML-first software using Oracle's Designer/2000 and Developer/2000 tools, because that was just the "right" way to do things. I've seen organizations destroy themselves in the morass of trying to do things with Enterprise Java Beans (EJB) because it was just the right way to do things. I've seen companies spend 10x more time getting their automated build processes and super duper full test coverage going than they spend trying to actually write software.
And I've seen companies navel-gaze over their processes and methodologies and buzzword compliance far more than they worry about meeting the needs of their customers. If you've ever worked for a company with a 15 year history of ironclad determination to use all of Microsoft's latest preferred data access methodologies, then you have first-hand knowledge of another example of this.
I'm not saying microservice architecture is a worthless fad, but I am saying that putting your business needs into the done-basket absolutely must be the first priority; never lose sight of it. I'm all for being wary of technical debt, but "technical debt" isn't nearly as dangerous as whatever you call that contagious illness that infects developers and makes them put architecture before business needs.
Where software is seen as a cost center and is actually developed 100% by contractors that fine-tune existing code bases across multiple years.
I guess there isn't better cure to those that tend to follow each fad that pops up.
It's important to find balance.
Maintainability and security are definitely business needs, but they're long-term instead of short-term, so it's easy for new companies to underinvest in them.
Plus the compulsory EJB pain. I found it quite liberating to be able to just abandon EJBs once it became clear what a disaster they were without having to do do the Emperor's New Clothes bit.
I kind of share the scepticism about shiny new paradigms now, but one does have to be mindful not to inadvertently reject good new things because they're new.
So when you don't have customers you get to anti-pattern and navel-gaze, because then, at least people are doing something.
That's why companies implement Agile. It's all about "solving business problems". No architecture, no design, no documentation (merely 'user stories'). Dump and re-write instead of maintain. All your problems solved!
A clean data model, one aligned with business needs, and sensitive to the nature of machines, makes the best architecture.
Far too many decisions are based on what is trendy and fashionable rather than what is right for the domain.
In most cases there others solving the same problem. If you are the team bringing a better solution to market faster, with better design, higher quality, less maintenance, lower infrastructure costs, fewer developers - you win easier and earlier. But, that takes some dabbling and experimenting.
In the end, it's really about working with high quality people. A team of focused, pragmatic, informed, and disciplined people can take the right new tool and become an order of magnitude better. And, they can take the wrong new tool and know when to shelve it quickly.
The winners in this game aren't writing COBOL against mainframes - even though that would probably yield a sufficient solution eventually.
I think this is missing the point. It isn't just about decentralizing services, it is also about allowing you to decentralize teams and decision making. With very large teams working on a single monolith, features that are complete often cannot be deployed because of the larger organizational overhead of planning and coordinating a release. The higher coupling in most monoliths also restricts a teams ability to try new things and take risks.
If performance of technology is the only metric that matters to you, then yes, microservices are probably a horrible idea. If you are having difficulty scaling your teams, then it might be worth looking at.
What I think the microservice architecture does is force some measure of good systems engineering on a team. You can't design a microservice application without putting effort into deciding how to divide the functionality and defining how those functional blocks will work with each other. You can get away with not doing that (until it's too late) when designing a monolithic application.
Of course if you have tons of cash and resources to throw at it then that isolation could be beneficial. But is it worth it? I have no idea.
That is what I was referring to when I mentioned nebulous APIs. In a properly-designed system, each component (be it a module of a monolithic system or service in a microservice system) should be a black box with known APIs and no side effects. Part of defining the API is defining the acceptable values that can come across the wire. If designed correctly, breaking another service should not be possible without an API change. An API change should be something signed off by systems engineering and communicated to all affected teams before it happens.
Done right, this is a fairly heavyweight process and so, as others have said, is only something that should be done when a project reaches a certain level of complexity (both code and team).
If your need is respond quickly to changing requirements this setup falls flat.
Also, if it's being done right, you rarely "break" an API in a single blow. You add new functionality, possibly via a new API. You give your consumers ample time to migrate. Only then do you remove the old API. One of my old teams had a hard requirement that all APIs had to be backwards compatible for 2 versions, which was essentially a year. We got very good at adding functionality without any impact on consumers that had not upgraded yet.
* Adding a new method to the API does not break the API.
* In certain language/middleware combinations, adding new properties to a parameter object does not break the API.
If we have a method that needs to change, we add a new method. New clients use the new method, and we support old clients until they are retired. When using JAX-RS, we can add new properties to JSON objects provided they are annotated correctly.
If you're constantly changing/breaking the API then that's just a monolith in micro-service clothing. But the system can be simpler and you can make more significant changes quicker.
Our teams really have no limits on the changes they can do. If the change really is so complex that it requires changing every system, then we just build another system with zero dependencies. In fact we did that once. Got it working really fast. Usually branch-by-abstraction resolves any big issues, while updated method objects or just new methods handle most "changes".
Granted, this is with several hundred developers across multiple engineering departments. YMMV.
Integration testing is already complex in these environments, regardless of whether the components are monoliths or microservices. BDD style integration tests should be almost equivalent, regardless of the architecture, since they are driven by the business domain.
"With well-defined interfaces and properly-sectioned functional components teams can work just as independently on a monolithic application as they can on a microservice application"
Basically if you build a monolith such that it is not a monolith things are ok. This doesn't happen in my experience.
Speaking first hand, working with 40 engineers on a 500k+ line codebase. The monolith is as robust as it's weakest component.
There are secondary organizational effects that come into play as well with monolith architecture that need to be mitigated. Because there's a single monolithic platform, the org believes it can throw any set of engineers at a problem in the monolith and get the same result. This largely gets down to what philosophies you buy into, but I personally believe long lived teams with long lived missions are better capable to innovate (team builds depth of expertise in a capability vs breadth). There are plenty of ways to do anything wrong of course.
More so, as the team scales to more engineers, the monolithic architecture re-enforces the monolithic engineering org which means an increase in communication overhead.
My experience is that by creating a monolith you create one way to do things that leaves the organization less adaptable to change to implementing a new technology or pattern.
I think it's you who's missing the point. That right there is why.
Turning your monolith into a micro-service architecture should add up to basically: - Wrap your internal modules in your favourite form of RPC - Replace the modules with stubs that call the RPC
If you need more work than that, the problem with your application isn't being a monolith, it's having functionality be way too tightly coupled.
If turning your application into a micro-service architecture actually _is_ that simple, then you already have well-delineated modules that your teams can focus on. Deployment is just about integrating the latest stable version of each module, and the decision process within each team needs only respect the contract around the interface they provide -- same as a micro-service.
Saying you need a micro-service architecture to keep a sane internal structure to your application is a symptom that you need to review your engineering practices, because people aren't respecting the interfaces, and it's throwing out the baby with the bathwater.
The network is not reliable.
My experience has been that lots of different people will touch a codebase over the lifetime of a piece of software, and they will have varying skill levels. Even if everyone on the team right now will avoid creating a ball of spaghetti, at some point everyone involved will be gone. Patches and new functionality end up getting bolted on and added to an ever growing list of technical debt (we'll get to it, we swear). 10 years down the line no one can maintain it because no one knows what the interactions are across the system.
Splitting code into separate services/codebases doesn't totally prevent this from happening, but it does make it a lot harder. First, adding coupling is no longer the path of least resistance when you "need to delivery yesterday". Second, each service is significantly easier to grok than a single legacy monolith, and therefore easier to refactor back to sanity.
Vaguely related, I also really like how microservices offer a non-insane approach to modernizing a large system. A lot of my clients have large highly-coupled mainframe applications written in Smalltalk/COBOL/etc. They want to migrate to a "modern" language (ie one with a workforce). Big bang migrations almost never work, so we end up carving off functionality, re-implementing it in the new framework, and then integrating it with the monolith via messaging/services. Microservices make it easier to pull out one piece and rewrite it, without having to also touch every client.
I wouldn't be so sure. More often than not, it's cheaper to set up a new module/library than it is to setup a whole new service, and I've seen plenty of cases where people just tacked functionality on to the handiest service because they needed to get shit done Right Now, and setting up a new service was a waste of time.
Here are some examples:
1. A chat service (race conditions, synchronization issues between the game server and chat service that had to be debugged using sequence diagrams)
2. A payment service that handled Facebook, PayPal, and other payment methods. (race conditions, complicated integration, difficulty upgrading for different products, complicated code base to handle multiple use cases)
3. An authentication service (complicated protocol, hard to integrate)
4. A worker service for handling bcrypt (because blocking a single threaded server for 0.5 seconds is not acceptable, race conditions)
5. A tracking service (could have probably just been a library you included instead of hitting an API)
Core issues:
- Race conditions
- Synchronization issues
- Complicated to upgrade/maintain for multiple products
- Become too generic and solve multiple problems (services tend to be used for multiple products in the company)
- Extremely difficult to debug
- Complicated error handling for when a service is not reachable
pro-tip: don't share databases between multiple products for each service, deploy a new service for every product with its own database.
I think I'm scratching the surface here, but I would be really careful with introducing this kind of architecture when you can do it all in one server.
Micro services are hard to write, so don't write them unless you absolutely have to.
Monoliths have the advantage of being able to trivially take out a global lock.
Removing global locks from a monolith to migrate to a micro-service deployment adds a ton of design and engineering cost.
The core banking services that actually move money around are isolated from the more "fluffy" customer facing ones that we want to be able to push updates to several times a day. We have to have incredibly rigorous procedures for updating services that control money, if we had a monolithic architecture then these procedures would have to be used even if the change was simply cosmetic.
What I'm finding, however, is that many of these authors have such a broad understanding of microservices are that they miss any benefit. Then, of course, they complain about there not being any benefit, natch.
I suspect -- and what I have feared -- is that the term "microservices" has been co-opted and rebranded by a variety of vendors and proponents. The goal here is to sell products and services, not necessarily solve problems.
There is certainly a huge wheel of hype in the technology world, where things become cool, then old hat, then abused, then nobody does them anymore, then they return under a new buzzword.
Having said that, in the future I'm going to always make a point of defining exactly what I mean instead of just using the term "microservice". My current definition is something like this:
- Pure FP
- Unix Philosophy
- Each microservice has less than 150LOC
- Common code, types, and persistence functions are moved to shared libraries to reduce interop concerns
Not sure if this makes a difference in the discussion, but I know that it will help me keep straight whether various authors actually have quibbles with microservices -- or are just re-applying their pre-existing OO thinking to a place where it doesn't necessarily map so well.
Sounds like a nightmare, you know how much instances / vms you need if you spawn one thing for 150LOC?
This is illustrative of the point I was making.
One of the big benefits of microservices is the total decoupling of the code from the underlying hardware. Things are glued together by Ops, not by code. So you might have all the microservices in one VM -- or spread copies of one critical microservice across a thousand vms. It's all configurable.
Properly designed, the execution model is orthogonal and the service itself doesn't care if it's invoked in-process or not. It delegates that to some other manager, which probably has a service interface of its own.
It's about sane abstract system boundaries, not mapping to real world facilities.
Take a large monolithic app or framework. What I've found is that if a problem is properly coded, while you may have tens or hundreds of thousands of LOC, the actual code doing the work is quite small, on the order of hundreds or thousands of LOC. The rest of it is all "wiring".
Moving to pure FP means that a lot of the code structure of OOP disappears and you're left with just the critical pieces. This consists of composed functions performing translations on immutable chunks of data. Your functions become microservices and the composition of functions, scheduling, and moving around of data become Net/Dev Ops.
There are many ways to fall off the path here. You start using mutable data, you start associating services with business domains, you start coupling microservices together more tightly than necessary -- there's a ton of ways you can accidentally screw up, and then you're probably better off with a monolithic app.
Microservices should be like ls, cat, or chmod -- small pieces of composable functions that run directly in the O/S. They don't blow up the system when they fail, they do one thing and only one thing, they're configurable using command-line switches, they're transport-independent, and so on.
$ wc -l src/ls.c
4980 src/ls.c
768 src/cat.c
570 src/chmod.cThe more command line switches the more code paths. The more code paths the less of doing one thing and only one thing.
I don't disagree with everything he's said, but what contributions has he made that makes him so much of an authority on how I write code?
He never said he was an authority...
He's a self-proclaimed "expert" and a "leader". Yes, he's basically saying right there that he is an authority on writing code.
That being said, I've read some of his Refactoring book awhile ago, and it's mostly good advice from what I remember.
If you have few developers most of the time it's a lot more efficient to have a modular monolith than it is to have 50 entirely different rocks.
Decentralizing teams, decision making, and allowing increased independence of agency within a larger org is great. Trying to do the same thing when you only have 5 people is usually borderline crazy.
Yes microservices (or simply services) are very good at large scale applications but they are simply an overkill for any startup project. Why not concentrate your efforts on selling your product first vs building the perfect infrastructure that no one uses? When time comes to upgrade - well upgrade. Amazon is a good example of a large company making the move from monoliths to services and they executed well. You can also do the same thing when it is absolutely required. You know, refactoring!
Microservices are a gigantic pain in the ass. They increase operational complexity significantly, and they will require you to make a lot of investments in building/buying/implementing infrastructure around things like logging, monitoring and config management. Your number of possible variables in QA explodes exponentially, and suddenly you have to worry about things like API versioning.
If all of this sounds easier than dealing with the tech / organizational debt in your monolith, then microservices may be right for you. Otherwise, save yourself the trouble and focus on scaling your business instead of your technology platform.
We have a monolithic application (well, an application composed of many libraries that have a dependency tree, but live in the same process) that contains individually deployable services that talk to one another over RPC. So when you're patching part of the codebase, you can deploy a branch to the servers that are executing the code you've just changed, thus no need to restart the whole thing. But you still have the benefits of shared libraries, moving functionality between services, collapsing service calls that are no longer performant, adding new calls temporarily, etc.
In this model, you can introduce and remove service calls when it's useful, as opposed to as a result of how the code was put together. You will often do this for performance reasons, but just as often you'll do it for deployment flexibility. When you want to prototype something new, you can create a new service, branch the codebase, and have it talk to the rest of the infrastructure via service calls. When you're done you can keep it as a service, or, just as often, you can collapse the service back into other services so we don't have excessive RPC calls.
When would we consider moving to services backed by different codebases? If our company were to grow so large that highly disparate teams would want to manage their own dependency trees--that's around when it makes sense to me to go microservice. (I not sure I'd call it microservices then, more SOA).
I'm not saying micro-services are a fad, because they are great for isolating domains and degradation of service, but that comes with a high amount of boilerplate code for remote invocation and serialization.
I would personally say, if your project has not reached 10,000 lines of code do not go that way. also if you can solve the problem at hand with a cluster of stateless monoliths, keep it simple.
Dear friends, keep at hand the Occam's razor
So build a distributed PaaS with hot-deploy, then a distributed HTTP database; then you can use microservices without any of the problems mentioned (complexity becomes a non problem since everything is using the same complexity and it very quickly becomes bug free, overhead is removed by PaaS)!
The important step with microSOA is that each developer can choose his tools. That trumps everything else.