Hacker News Reader

Comment by Animats | Hacker News Reader

Rule of thumb: if you have two "service" talking to the same set of data in the same general-purpose datastore (i.e., not pub-sub, not opposite ends of a job queue), they're the same service.

Why? The whole point of a general-purpose database is to allow multiple applications to use the same data. Consider an online ordering system for merchandise. There's an service for checking product availability. There's a service for building a shopping cart. There's checkout and payment, the only one that faces the outside world that needs high security. There's customer order tracking. On the back end, there are various services which deal with fulfillment, shipping, accounting, and reordering. There may also be customer-relationship systems which can read order data for marketing purposes. Each of those functions can be worked on independently.

View on HN

In this case I think it's proper to consider the database as a service in its own right. This is the way it always used to be done, and there are significant advantages, such as being able to focus on safeguarding the data, and leveraging ACID and constraints/stored procs to make the application code less error-prone.

The downside is you have a potentially hard scalability ceiling, and you have coupling of every downstream service to the schema which means all teams have to coordinate with the DB team for schema changes.

I think startups now always go the vertical services route without thinking too hard about it because A) they all aspire to be Amazon/Facebook/Google even though 99.99% will never face that scale and B) resume-driven development.

A variation on this would be to put the database behind a service that abstracted over the schema, though that only works for basic CRUD queries and not complex aggregations. This service would probably evolve from the monolith.

You mean like an API?

It sounds like you're describing ORM-as-API. Which, if you find you need it, great, but I wouldn't try to fool myself that introducing a mediating process like that is all that different from what the language/framework likely already provides (contemporary development frameworks are common). It's just ("just") abstracted to operate on network basis. Actually, what you describe sounds like a reduction in functionality from a non-API model.

Hopefully that parses as English, it's still morning for me.

As I point out occasionally, Wikipedia is MariaDB front-ended by various caching systems. There's some auxiliary stuff for logging and searching, but its synchronization is not mission critical. Wikipedia is the ninth most popular site on the web. You're probably not going to be bigger than that.

Wikipedia is also massively cacheable, much more so than almost any other application.

> The downside is you have a potentially hard scalability ceiling

Which 99.9% of projects will never ever hit, and those that do will by that point have become such an extreme scale that significant reengineering should be something that's happening anyway.

I couldn't agree more with your last paragraph.

This can't be said often enough: Scaling is a luxury problem!

It is far more likely that a company gets a "small", stable user base that provides enough income, than it is to become the next Google.

It doesn't make sense to apply all the complexity for large-scale problems when your user base isn't even small-scale. And if your monolith is structured in a remotely sane way, you can introduce microservices (or whatever will be best practice) later on.

On the other hand, it doesn't hurd to write your code with performance and scalability in mind. Neglecting this too hard may result in bad performance even for the first 10 users. But that's more in the sense of "keep it simple enough so it can be easily restructured later on". And in the sense of "measure performance to know your hot spots" ... and concentrate you efforts there.

The main problem is that micro service based architecture is sold as a silver bullet that takes care of all problems.

Knowledgeable and experienced people know that there is no free lunch, but not every opiniated voice isn't necessarily so, and refuting bullshit takes ten times more effort than producing it, so objections might be handwaved away as backwardness if there is a strong force for a modern architecture.

I mean, more of the same old shit isn't exactly an easy sell either, even if every old bad choice could be rationalized at the time...

> On the other hand, it doesn't hurt to write your code with performance and scalability in mind. Neglecting this too hard may result in bad performance even for the first 10 users.

Absolutely. But the architectural choices you make for making the first 1000 users fast are often the complete opposite of ones you'd have to make to make it "scale" to 100k users.

Service/microservice architectures that I've seen are, from an absolute point of view, hideously inefficient, simply because the component that needs data isn't able to ask for the data from the other component (way across the other side of the architecture) in a precise enough fashion. Cue reams of data being sent to you that you don't need because you need to figure out one small aspect of it. And of course, jsonifying it all and de-jsonifying it a few times for good measure. All this with no transaction safety.

Now, if you're operating at the kind of scale where you know that these specific calls are now part of your business critical requirements, you can spend the effort optimizing the hell out of them and reap the gain of horizontal scalability. Of course, that comes at the expense of flexibility.

> resume-driven development

To be fair, until companies are willing to pay a premium (1) for working on bad-for-resume projects, this is very rational behavior on the part of the technical team.

1) That is, job listings can't have merely "competitive" or "market rate" salary expectations.

A huge point of microservices is to create code that is independent of eachother. Your database backend is tightly coupling what should be completely independent services. Microservices were designed to scale your engineering departments just as much (if not more than) your performance. You are throwing away the major point of why microservices were created in the first place when you do what you're doing: so teams in large organizations can do independent releases. That you cannot see this as a major benefit makes me think microservices are a poor choice for your use case.

> You are throwing away the major point of why microservices were created in the first place when you do what you're doing: so teams in large organizations can do independent releases.

Let's ammend this: so teams in large organizations +{that lack both a coherent architecture and the ability to devise one} can do independent releases +{by adoptiong the no-architecture architecture}.

"teams in large organizations"

Could be simplified to just "teams.

These days from what I can see teams in organizations with no ability to devise a sane architecture usually take the microservice route. Management loves this new buzzword. To them it means problems go away.

Atlassian started rearchitecturing around services in 2013. They were 1500ppl at the time. Confluence and JIRA were not only monoliths, but non-multitenant, so they had to have one instance per customer (700Mb RAM JIRA, 700Mb Confluence, a dozen gigs on SSD). The worst was restarting upon upgrades: Easily 3-5 minutes per instance, which, at scale, was a huge burden.

After rearchitecting around services, pieces could be restarted and upgraded independently. As a customer, we didn't notice differences in the UI (e.g. the file storage on AWS).

Epilogue: They had a shared login system, multi tenant and all... which they recently replaced by a third party. Proof that services are replaceable, but also an acknowledgment that simple critical services can be hard at scale.

I personally believe that Atlassian switched to services at the right time, when it's hard to coordinate teams of thousands working on a dozen products, and when the monolithic approach was way past refactoring date ;)

On the flip side, cloud hosted jira is soooo slow now.

A bit late to answer, but by their EULA you are not allowed to talk about the performance of their products.

Yes, they did that!

And Jira still can't handle email in a sane fashion...

I wouldn't look to Atlassian for anything regarding engineering practices.

> Confluence and JIRA were not only monoliths, but non-multitenant, so they had to have one instance per customer (700Mb RAM JIRA, 700Mb Confluence, a dozen gigs on SSD).

I don't understand how you engineer even a monolith like that in the first place.

Rather a late reply but Service Oriented Architecture is in fact an architectural approach. "Micro" services deem even the burdern of devising a coherent "service" as too great a burden to tackle.

can you describe a coherent architecture that will turn microservices uneeded?

Missed this but in case you read this, well that depends on the purpose of the system, but in most cases modular schemas and allowing for loose coupling via data/message bus does the job.

If that needs to be scaled out to many services, then you also want to scale out the data model. Meaning, events in a commit log, and services that subscribe to that commit log to create their own schemas that are tuned for their own needs. The more I think about commit logs, the more I think giant monolithic database schemas are just kind of weird - a strange compromise between history and state.

That's an interesting idea. If I understand correctly, you're suggesting that services simply broadcast the data mutations they are going to perform. Then, any other services that wish to use that data can filter those broadcasted mutations down to what is relevant to the other services, and store the data in a way that is efficient for what the other services want to do.

Are there any examples of things architected in that way?

Doesn't it also run into the problem of needing to version the events that you broadcast?

Event Sourcing [0] & CQRS [1] come to mind. I'm also a fan of Bottled Water [2] that turns Postgres ops into a stream of events that you can then push to other services.

[0] https://martinfowler.com/eaaDev/EventSourcing.html

[1] https://martinfowler.com/bliki/CQRS.html

[2] https://github.com/confluentinc/bottledwater-pg

It's hilarious to me that that came up, because that was a big part of the 6 month delay.

Any particular one or all three of them?

It's the Kafka cool-aid being regurgitated. Kafka is the pipe that you need. Micro-services provide APIs to state that they manage that they get from Kafka. That all scales out and services have nice independent failure modes (as long as Kafka doesn't fail). What it doesn't consider, however, is if you have to perform atomic operations across micro-services (you can't easily do this with existing technology). And, yes, you would like exactly-once processing of events, which is non-trivial - even with the new exactly-once message reception feature in Kafka.

If your services are completely separable, this approach is great. Otherwise, monolith it.

this sounds like an absolutely enormous pain in the arse (cognitive overhead if you prefer.) what, specifically, is the bigger pain in the arse out there that one needs all this in order to avoid? it must be pretty awful.

also, what happens down the line when you suddenly need a service which is contingent on the data stored in multiple other services? do you add another synchronisation layer over the top?

Whether they can be worked on independently is not the criteria that matters (and in point of fact, its a false assertion in this case if they're using the same database, as neither can iterate on the schema of the data completely independently of the other). What matters is, can I take this service completely offline and have the other still function? If the share a datastore, then the answer is no.

All services depend on a router, so you can't take any one service _completely_ offline, so they are all part of one service.

You presumed before you even started design that there would be such a thing as an order, which would be implicated in shopping, payment, and tracking.

There could also be separate order, invoice, and delivery.

Essentially your question is: what if my models are highly overloaded? Won't they need to be used in multiple places?

And of course the answer is yes. But if you are using the same model in dramatically different contexts you probably have a problem of another sort. This is the dreaded "One Object To Rule Them All" OO anti-pattern.

And one day you realize you have to change your data model, and absolutely all your application are coupled to your whole database. Oups!

You can add a column to an SQL database without difficulty. If you're using one of those "turn database rows into objects" packages, you may have problems, but if you use SQL properly, it's fine. Components that are only data consumers will not even see the new column.

There is a solution for this problem: I use a monolith that make 100% of my sql queries using LINQ (including massive updates and so). Since Linq is strongly typed you can rename/remove/split tables and columns, fix compilation issues, create a migration and deploy the new monolith. We create an average of 5 migrations per day in a project with around 300 table for 2 years and no signs of an unmantainable mud so far.

Disclaimer: I'm the main developer behind Signum Framework

Just inherited exactly that.

The final cherry is that it uses zero joins, it's all done in code.

It's going to be a bastard to unfuck that mess.

Reply on news.ycombinator.com