Modular interfaces are great, but why require communication over those interfaces to go via network traffic? Can't you just use a library/package?
1. Multiple languages/tech stacks. Have something like protobufs and now any language can "import" your library instead of it just being things that can handle cffi.
2. State management. For example, redis couldn't be an in-process library. There are in-process kv stores but that's not the idea of redis.
3. Transparent monitoring for all languages. I can passively detect all retries, latencies, request counts, etc since there is a protocol.
4. Centralized fixes. Similarly to dynamic linking I can now update a single component and "fix the world" rather than redeploying every binary.
5. Canaries: I can deploy version N+1 to a mirror of production traffic and make sure it doesn't explode. Then I can route 1%, 10%, 50%, 100% of traffic and check those metrics as well.
6. If something goes horribly wrong I can rewrite the entire service for performance or maintenance or whatever within a tightly bounded amount of time. Discord has many blog posts about doing this for some of their services.
Great talk: https://www.youtube.com/watch?v=-UKEPd2ipEk
This is what I'm excited about--enabling a team to use the most powerful tools that address their own problems, rather than whichever Blub is approachable enough for the entire staff.
Say you run a factory, and everyone needs a hammer. When there's only two teams, two kinds of hammer is fine. But fast forward to 10 teams. Can you always get all of those hammers when you need them? What if team #3 needs extra support? You need to find a new engineer who knows hammer #3, or take an engineer off of team #6 and train them on how to use hammer #3. And if all the hammers need a slight modification specific to that company, you need to make 10 different modifications. And how will all the teams benefit from shared hammer tips-n-tricks if they all use different hammers? There's a lot more overhead than you think at first.
Unless you have the specific need of that piece of code not being in the same process/container/computer/network.
Unless your linking time is prohibitively high (but even very large software takes minutes, not hours), the benefit to compile time should be the same. But maybe "redeploying" is more disruptive (I don't know, I haven't done web development for something approaching 10 years now), and depending on the language, problems in one part can crash the whole process. (But depending on the actual process model used, shouldn't it just restart? Unless the language is not sufficiently memory safe, e.g. it's C, and you get way worse problems with memory smashers.)
Again, my, and likely OPs point, was that distinct, modular development (and for example the mentioned low compile times) is not contingent on separate services and IPC or even RPC, which come with a whole other can of worms.
(BTW your scope of shared library is also pretty narrow. For persistence for example, SQLite is essentially a persistence layer that is a shared library. And some libraries do certainly spawn threads for their work. But that’s not really the point.)
SQLite is a great library, but that's more of a business-agnostic tool. How do you encapsulate a dedicated portion of business logic, let's say, managing image datasets for building machine learning models. e.g. I want some apis for basic CRUD on datasets, i want to automatically create new versions of existing datasets as images get published, i want to generate stats in the background for these datasets, i want a ui to review and manage datasets for model building, and then i want all this packaged in a single library?
Also, to clarify, I'm not playing a game here. I don't care about theoreticals, I want to hear about how people have actually solved their problems, so I can then evaluate and use those approaches. So I'm trying to get at something a little bit more specific and less handwavy.
In a monolithic architecture these would run in the same address space, be separated into modules by the language's own means of abstraction, and the communication between them would be done by function or method calls.
I don't see what HTML files have to do with it, what do you think microservices are?
You would have a hard time to make a resource efficient distributed in-process key-value store.
You could have an in-process library but this would cause your two programs to have either:
1. Store different data
2. All writes would need to be distributed to all processes (looping back to needing a network connection).
Yep, exactly. I've built more than one microservice in my time that loads up a few GB of data into RAM and efficiently queries it for a lot of services. At that point the line between DB and service gets blurry.
But, essentially, the TCP connection between your services isn't necessarily a big issue. If there is some logic/operation/access that is complex it makes sense to chuck a language-independent interface in front of it for most use cases. There are some where performance trumps maintenance but where it doesn't I think this abstraction is helpful.