Hacker News Reader

Comment by therealdrag0 | Hacker News Reader

I used to be monolith-curious, but what sold me on micro-services is the distribution of risk. When you work for a company where uptime matters having a regression that takes down everything is not acceptable. Simply using separate services greatly reduces the chances of a full outage and justifies all other overhead.

View on HN

Why would using microservices reduce the chance of outages? If you break a microservice that is vital for the system, you are as screwed as with a monolyth.

Sure, but not all micro-services are vital. If your "email report" service has a memory leak (or many other noisy-neighbor issues) and is in a crash loop then that wont take down the "search service" or the "auth service", etc. Many other user paths will remain active and usable. It compartmentalizes risk.

Proper design in a monolith would also protect you from failures of non-vital services (e.g. through exception capture).

So it seems like we’re trying to compensate bad design with microservices. It’s orthogonal IMO.

How does exception capture protect from all failures? The most obvious one I don't see it relating to is resource utilization, CPU, memory, threadpools, db connection pools, etc etc.

> we’re trying to compensate bad design

No I think we're trying to compensate for developer mistakes and naivety. When you have dozens to hundreds of devs working on an application many of them are juniors and all of them are human and impactful mistakes happen. Just catching the right exceptions and handling them the right way does not protect against devs not catching the right exceptions and not handling them the right way, but microservices does.

Maybe you call that compensating for bad design, which is fair and in that case yes it is! And that compensation helps a large team move faster without perfecting design on every change.

With microservices you have to have a tradeoff - a monolith is inherently more testable at integration level, than a microservice based architecture.

There's a significant overhead to build and run tests at API level, that includes API versioning... and there's less of a need to version API inside a monolith.

You have fifty (or 10,000) servers running your critical microservice in multiple AZs. You start a deployment to a single host. The shit hits the fan. You rollback that one host. If it looks fine, you leave it running for a few hours while various canaries and integration tests all hit it. If no red flags occur, you deploy another two, etc. You deploy to different AZs on different days. You can fail over to your critical service in different AZs because you previously ensured that the AZs are scaled so that they can handle that influx of traffic (didn't you?). You've tested that.

And that is if it makes it to production. Here is your fleet of test hosts using production data and being verified against the output of production servers.

If you have a truly modularized monolith, you can have a directed graph of dependent libraries, the leaves of which are different services that can start up. You can individually deploy leaf services and only their dependent code will go out. You can then rationalize which services can go down based on their dependency tree. If email is close to a root library, then yes a regression in it could bring everything down. If email is a leaf service, its code won’t even be deployed to most of the parallel services.

You can then have a pretty flexible trade off between the convenience of having email be a rooted library against the trade off of keeping it a lead service (the implication being that leaf services can talk to one another over the network via service stubs, rest, what have you).

This is SOA (Service Oriented Architecture), which should be considered in the midst of the microservice / monolith conversation.

> Simply using separate services greatly reduces the chances of a full outage and justifies all other overhead.

or maybe run redundant monolith fail over servers. should work the same as micro services.

then you need to hotfix you need to re build a giant monolith that probably has thousands of tests and a 20-30 minute regression suite easily.

I have seen exactly this. Waiting for a 60+ min CI build during an outage is not a good look.

However, there's another "bonus here" is that you have integration tests that have a better coverage.

Microservices don't make builds radically faster for the majority. People still split systems into larger services.

How do microservices help here? You can deploy a monolith 10 times and have the same risk distribution.

It's not about replication, it's about compartmentalizing code changes / bugs / resource utilization. If you deploy a bug that causes a service crash loop or resource exhaustion isolating that to a small service reduces impact to other services. And if that service isn't core to app then the app can still function.

Reply on news.ycombinator.com