The Cracking Monolith: Forces That Call for Microservices | Hacker News Reader

The Cracking Monolith: Forces That Call for Microservices

semaphoreci.com

107 pointsmarkoa52 comments

I find this article lacking, since it paints a too simplified brush over monoliths. Some missing points:

"Monolith", at least if you define it as a single process rather than a network of (micro)services, can still be modular in nature. Using a flexible core, you can build your application as a combination of modules, rather than as a single large codebase.

They also don't have to be a SPOF: in our architecture, we deploy monoliths, one for each client (ie, company, not user), while keeping them isolated from each other. This allows us to scale pretty much indefinitely using a simple architecture. We don't have a huge sharded database, but small bundles of databases. We don't have datacenter-wide load balancers, rather we allocate clients to certain groups of nodes and use DNS to point them there.

This works because our clients don't really need to interact with each other, unlike say, social network users. Do yours? If not, why are you building a huge and complex mesh of services?

We have the same architecture. It's really versatile, simple to administer, and with the help of scripts and automation a new client can me mounted in less than 3 minutes.

Isn't it hard to maintain all the servers? How do you upgrade your application or your database?

This is why you use configuration management tools. We have a similar architecture, with dozens of deployments for customers; upgrades are just a single command that changes some configuration, and puppet does the rest. Until something goes wrong, 100 servers are no harder to upgrade than 1.

We automate it :) There's an "inventory" of the current instances, and then using configuration management (currently Ansible) we can deploy a new version (running migrations, starting new processes and stop old ones, etc) for all or any subset of our instances with a single command.

It does sound like upgrading all the customer servers would be a hassle, but that might be a feature in some domains - if customers have process / regulatory / training constraints and don't want to upgrade promptly, for instance.

Use caution. The multi-instance monolith approach can run into a nasty obstacle at a certain point when you want the business a very, very large customer who does need all that interaction within their own service. If you've been doing your scaling on shared resources the whole time, then you'll be far better equipped to handle it and the code is less likely to be broken.

(Also, even short of complete scalability failure, things like maintaining proper redundancy on a per-customer basis can be a major pain point.)

Of course the only real way to get these tradeoffs right is a very good understanding of your problem space.

Largest installation we were involved with (though not me personally) was a 250k user base, for the Ministry of Education in my country.

The monolith is stateless, so it's pretty easy to scale. The painpoint is Postgres, but I think people underestimate what real hardware in a small cluster can handle.

I think GP's point is more along the lines of When the Ministry of Education and the Ministry of Whatever are both customers (separate instances) and want cross instance reporting.

We had this at one of my previous places, exactly the same setup. Identical monoliths * shopping malls, the issue is that Companies own more than one shopping mall and they really want cross mall data, something that was extremely painful to add in at a later date.

Except that in real world you can't automagically create cross report from separately defined data structures.

If you want cross report you will define a super-data structure able to talk to both monoliths. They incidentally become "microservices" but at a higher level.

I agree with Twisell. The solution is to create a super data-structure so this really is an architecture and abstraction issue - ie. design things in a way such that these structures can be added in an incremental and painless way when the need appears.

Only a minority of people are good at that. I'd also note that there is a correlation between the importance of such needs and the ability to anticipate them at the initial design stage. (ie. is it reasonable to be surprised by the fact that a holding owning several supermarkets asks for consolidated reporting at some point ?).

If you're good at abstraction, modularity and composability, there really isn't any gotcha to fear. Stuff don't just come out of the blue.

What exactly is a super data structure?

How do you create a data structure for a requirement that doesn't exist at the beginning of the product?

Most of the time when I've seen someone design data structures in a way that attempts to design for unknown or predicted future requirements, the design ultimately becomes over engineered and a nightmare to maintain or expand in ways the actual product requires down the road.

I'll always pick the concrete implementation as long as future requirements are unsure.

I don't know how you get good at magically guessing how your product needs to evolve, e.g., what new industries is your sales team going to crack into, what direction does marketing want to take your product, etc - the only way you could be "good" at that is when the direction is extremely obvious or you have unilateral decision making on every point of your road map.

> cross instance reporting

Isn't a DWH the best solution for this?

I evaluated another system where we needed something similar. Adding cross data reporting was not hard. Combining whole installations was a bit harder but it was probably not harder to solve it when needed instead of solving it early

Author without even knowing it described common microservice architecture problems that are faced in front-end. (Because in microservices front-end is consuming API and is usually a SPA)

- The single point of fragility

Yeah, if you have an js error in your syntax the whole front-end application will not work.

- Slow CI

Actually the front-end nowadays is the bottleneck of any CI if it is using React/Angular2 because of all the tree-shaking and optimizations. It can take a good 8 minutes to build, optimize, tree-shake, AOT etc. a modern SPA application.

- Slow development cycles

This has nothing to do with monoliths whatsoever. If your codebase is crap it doesn't matter if it's microservices or monolith.

Anyway, I digress since this is a textbook Mary blog article.

- Yeah, if you have an js error in your syntax the whole front-end application will not work.

How is this possible that you compiled js code with an error during packaging? Or you don't use packaging/analyzers tools?

If you reference a null object, an exception is thrown.

"Textbook Mary blog article"?

It's taken from "Disrupted" book.

It's basically a blog post that is pure fluff to generate inbound traffic with no real substance.

Think "10 ways to increase your Node.JS performance", "3 reasons why your AD Campaign is not working" etc..

Thanks, useful phrase!

> It can take a good 8 minutes to build, optimize, tree-shake, AOT etc. a modern SPA application.

Can't you break your SPA into modules that are versioned and build independently to help solve this problem? With additional infrastructure it can also help avoid loading unnecessary code on pages that don't need it, etc.

I have not described nor have hands-on experience with all-in approach to microservices. In our practice all user-facing stuff comes from a monolith. I'd probably argue that fragmenting UI-serving backend so much is not necessary for 99% of teams.

I would like to argue that a microservice architecture is frequently a response to the scalability limits of traditional databases. Decomposing a monolith into multiple services (with each service interacting with only its own database) is essentially a form of vertical partitioning. It's a way to delay sharding.

Microservices are nice, but they seem like a heavyweight way to "force" module boundaries to exist. Potentially, if database technologies improve, there might not be a need to split up a monolith until much later on.

the danger in my view is that it is seen as a binary choice - which, if you go all in on microservices, just shifts from one problem to another - they need to be seen as just another tool for architecting and scaling

there is nothing wrong with building a monolith so long as you gave some thought to how it might get partitioned later

once you've built a few I think you start to see where those pressure points will be and microservices are a good pattern for scaling the functions that get all the traffic once it's working

I hear people talking about using only microservices to underpin a big app but I just can't imagine that is an efficient or sustainable approach for an app with any complexity if the core function spans across all those services - at the very least I can see macro changes becoming disproportionately more challenging and risky over time

> I hear people talking about using only microservices to underpin a big app but I just can't imagine that is an efficient or sustainable approach for an app with any complexity

Not having some coherent analytical approach probably isn't efficiently sustainable for complex systems, but if you have one, microservices for the whole system probably aren't a problem, since most system analysis approaches end up describing/modeling systems in a way that has a natural corresponds to a set of microservices wired together with a messaging bus.

> early adopters have been tech behemoths such as Amazon

Not true. Amazon is heavily service-oriented, but quite wary of micro services. You need the right size - microservices are often too slow and require a lot of work to debug.

The term micro was always left up to the implementer to define what exactly constitutes micro. There is no uniform standard, and no official definition of microservice, and indeed in my own short time dabbling in them, I have seen it vary from org to org.

The ultimate example I have seen, and used myself, is an AWS lambda function, but even then you talk to some "microservice experts" and they don't consider this to be the case.

hmmm i always thought there is clear definition. So microservice is :

A smaller problem domain. Built and deployed by itself, running in its own. Runs in its own process. Integrates via well-known interfaces. Owns its own data storage.

https://youtu.be/wgdBVIX9ifA?t=13m10s

What is micro services https://github.com/msjaiswal/book-notes/blob/master/Building...

* copied from stack overflow

I have a copy of the book you linked in the book-notes! I am excited to start reading it. It's funny you bring up that book because I arrived at my conclusion that how small "micro" is in microservice is based on my previous viewings of talks and articles by Sam Newman and Fowler. I think a lot of microservices are is just applied distributed versions of Enterprise Integration.

> microservices are often too slow and require a lot of work to debug

Interesting, can you share some example from your experience?

> Emergency-driven context switching: we may have begun working on a new feature, but an outage has just exposed a vulnerability in our system. So, healing it becomes a top priority, and the team needs to react and switch to solving that issue. By the time they return to the initial project, internal or external circumstances can change and reduce its impact, perhaps even make it obsolete. A badly designed distributed system can make this even worse — hence one of the requirements for making one is having solid design skills. However, if all code is part of a single runtime hitting one database, our options for avoiding contention and downtime are very limited.

Flying Spaghetti Monster, this has been my life for the last three months. React, react, react. Spend two days working on new features, spend the next three on emergency bugfixes for old releases, rinse and repeat.

I'm so ready for a vacation.

I'm not sure what we (bioxydyn) have done constitutes a microservice but we have split different parts of our business logic into separate executables that communicate using HTTP and a central data store. Our product is a data processing pipeline and we took this approach for several reasons which I may have missed in my skim read of the article. Firstly different parts of the pipeline have different resource requirements and would thus be hosted on different hardware configurations. Also parts of our pipeline lend themselves to different designs and languages (some Java, some C++) decisions which are also influenced by the availability of third-party libraries. Finally, by designing the overall system as a set of coordinated individual services we allow greater flexibility in our deployment model; allowing us to cope with some unknowns around scaling and regulations in different environments.

I'm no microservice expert but I'd guess the difference between your approach and microservices is that multiple instances of a microservice run concurrently, so they're inherently easier to scale?

A CI company that charges per box trying to persuade you to get more boxes.

Consider a contrary argument. If every app in the universe was composed of microservices that do one thing and test out in 5 seconds, then there'd be no need for CI, only monitoring & continuous deployment.

Because everyone knows that the "ops" part of devops is the part that is enjoyable and scales well. /s

On a more serious note, if that is the goal why not just write a monolith in Erlang? You can hot-patch functions, scale on multiple machines, and don't need to use http as IPC.

I wonder how much criticism and hatred this post would've received had it not used Rails as the example case. Monoliths exist in other stacks (obviously), but the Rails community always seem to take offense first. This is the first time I've seen such an opposition to micro services (maybe I'm not looking through forums hard enough).

> This is the first time I've seen such an opposition to micro services

Like anything, use the right tool for the job and weigh the trade offs

https://martinfowler.com/bliki/MonolithFirst.html

https://www.martinfowler.com/microservices/#when

Everyone is hating on this post -- for good reason -- but keep in mind that this is an inbound marketing post for a CI company. It's in their best interest to have you split your app into many standalone pieces which will each require an individual testing setup.

CI resource-wise, splitting things is actually more beneficial for customers than vendors. Gives more throughput.

Would a microservices architecture be feasible for a team of 3 developers?

It seems like only larger companies are adopting this architecture.

Another aspect to consider is, time taken to run regression and automation tests on the Monolith, by both devs & QAs. This affects frequent release of software considerably l.

That's because you're testing the monolith end-to-end, but testing the microservices individually. None of this is inevitable: you should still test the microservice network as a whole, and you can definitively test the components of a monolith individually.

Agree that the functionality needs to be tested in-whole. But the delta in time is very large when the feature level tests have to be run on the monolith & they can be avoided when they are tested on the micro service.

But where does the difference come from?

A "well-designed monolith's" code architecture would be very similar to microservices - with separated modules, but in-process function calls instead of remote API calls.

If you do large-scale integration tests that cross module boundaries in the monolith, then you should also have the same tests of the full deployment with microservices. If you test each microservice independently, then you could also run tests for each module independently.

My experience (sample size of 1) is that long-lived monoliths tend not to evolve along "well-designed" lines, but rather along "what can I possibly do to get this to work in the time I have?" lines. Further, it's an impossible hurdle to schedule big dig to make substantial architectural improvements on a large monolith, so refactorings and improvements tend to be shallow and localized.

A decade later, and you have a lot of business logic in stored procedures in your database and now hear grumbling from coders who don't know or don't want to learn SQL. (And, to be fair, it does make it harder to scale the engineering team if everyone needs to be proficient at SQL.)

Neither availability nor technical scaling concerns were the reason we started to transition away from our monolith. We were regularly hitting 4 nines of availability (measured by "orders coming in") and had scaled to well over $1BB in revenue essentially on a single database for most of the site functionality. Above average engineers and ops, and way above average DBA team was key to making that happen.

Running unit tests on only the parts of your monolith you touched should be fast. Running the full test suite will be slow, but then testing a full architecture made of lots of micro services, with all possible race conditions and other concurrency complications, is way more difficult, at least in my experience (and that is with few not-so-micro services).

surely the microservice set needs to be tested as a whole too - especially if the services are consumers of each other

this is the biggest argument against going large on using them in my view - you create a collection of ghettos with the intention of partitioning - but end up with spaghetti when it comes to maintenance and adding macro features later - especially as developers come and go and don't fully understand the architecture

I completely buy using them to scale very high load core functions or to separate isolated non-core functions - in moderation

Hi, suggest you to read on the test pyramid and the advantages micro services bring in to compliment it. Thanks

I think you mean that usually testing monoliths is easier (no network and so on) but I am not sure, can you clarify?

It should -- microservices require testing versioned interactions between services.

Let's say you're changing the behaviour of a feature in service A that is consumed by service B.

Service A is at version 15, service B is at version 42. You have to deploy service A version, but it has to have the new change but still also be compatible with service B v42 (because that's what's running). Then, when service B v43 is deployed, service A has to start serving the new functionality.

This gets exponentially more complicated with more services, more dependencies and more versions.

I'm working on a large system with is based on microservice architecture. We primarily use microservices to decompose the system into modules and enforce module boundaries. Things like scaling or performance is not so much of the problem, the monolyth would most probably work just as well.

Technically we write Spring Boot-based microservices which mainly exchange messages over RabbitMQ and in some rare cases call REST interfaces of each other.

Below a few problems I have with this architecture:

* Code duplication. For instance, we have to implement more-or-less the same DTO on both sending microservice as well as receiving microservice.

* Refactoring across microservices takes a lot of effort. In several cases we didn't cut in the right places and had to move functions across microservices. That was always quite difficult, and we also had to implement migration routinges (for databases as well as unprocessed messages which may be still in queues).

* Versioning and synchronization of messages. Our system receives events from the number of external data sources which trigger processing in the microservices and finally produce results. Since message exchange is asynchronous, older messages may overtake newer messages so we have to version them in order to not output obsolete results. Another problem is that messages travel different paths in the system, so it may happen that a microservice receives two messages which are based on partially obsolete data. Consider two sources A and B which produced events a0, a1, b0, b1. Later in the system a microservice receives two messages p and q based on p(a0, b1) and q(a1, b0) respectively. These messages are not compatible and cannot be processed together. But in order to detect this we have to track version of incoming events (a0, a1, b0, b1) all along the processing.

* A hell to debug. At the moment we have a network of around 40 microservices which process data from ca. 6 sources and feed 3-4 receiving systems. Microservices are quite complicated, the work with often incomplete or incorrect data, employ a lot of heuristics and make a number of assumption which are sometimes proven wrong. If any of the receiving systems report a problem we have pretty hard time identifying which of the few dozen microservices failed. Reasoning about asynchronous message processing is very difficult and approaches like remote debugging actually change the behaviour of the system.

* Tools. We employ maybe a dozen of different tools and custom scripts solely for the purpose of configuring, deploying and running our microservices. We'd most probably only needed a small part of them for a monolytic system.

From the other hand our system is a fourth attempt to solve a very difficult and important problem for our company, the problem which persisted for more than 30 years. And it is the first attempt which is so far successful.

Reply on news.ycombinator.com