Comment by kgeist | Hacker News Reader

Comment by kgeist | Hacker News Reader

Parent Full thread

Another team's product at our company runs inside a Kubernetes cluster which consists of lots of microservices. CPU's are at 80% all the time. Redis, RabbitMQ, MySQL, Elastic Search etc. There's no number crunching, mostly CRUD with some business logic on top. It requires a separate operations team to manage it (otherwise this goldberg machine easily explodes).

A few days ago I asked our ops team, what's our rps? ~500. Out of curiosity, I ran a few benchmarks to stress test my monolithic pet project in Go with a single DB for everything and it easily handled 15k/s on my home computer, too, without any special optimizations. I understand its load is totally different from our production server, and my benchmark is most likely flawed, but the order of magnitude of difference is interesting.

I was told the production cluster also has additional 500 rps just for communication between the microservices. I suspect there's just too much overhead in their setup: communication between services, serialization of data between different DBs, etc. If it was a single modular monolith with a single DB, I suspect it could be faster and easier to maintain... Another issue is that they also sell it as an on-premise solution. And with the complexity growing, it now often fails to run on clients' infra.

We had pretty much the same deal at my last job. We had somewhere around 50 services that ought to have been one. Batch processes would take hours as they made 10s of millions of internal API calls (which could also fail of course) instead of running for a minute or two just doing the work with a couple SQL queries. Who knows how many hours wasted debating the boundaries and contracts between services when those layers didn't need to exist in the first place. Millions a year spent on hardware. All in the name of "scalability" when we also had a few hundred rps, and the thing obviously had scaling issues exactly because it was being done that way. Very frustrating.

I bet those CPUs are mostly busy just de/serializing JSON, processing HTTP and TLS for all the various intra-service calls.

Not to disagree, but if your company has around 300,000 LoC in business logic and your pet project has 1,000 LoC, the processing times are going to be significantly different.

It has more business logic, true, but the average request is just the usual "retrieve an object from the DB, modify a few columns, and save back". Just like in my project. They have a higher LoC because they have more use cases, but a single use case alone is not very complex and is comparable to my project's code in its complexity. So I think it's fair to compare the two when it comes to rps.

But imagine how many billable hours were created with this thing

Unless it's a consultancy project it's less about billable hours and more about internal busywork and resume fodder.

In a ZIRP/startups environment this kind of BS was rewarded by the market and thus "engineers" would take any opportunity to build up such unproductive over-engineering skills to polish up their resume for the next opportunity.

At first I was "eh, if it runs okay and can just be scaled, it's fine as long as they know the limits, how it scales, and it's maintained".

> And with the complexity growing, it now often fails to run on clients' infra.

then I saw this and... yeah completely different beast

Reply on news.ycombinator.com