This is happening to me, its driving me nuts. Any advice on how to get out of it?
1) A direct method invocation resolved within the same L1 cache contents.
or
2) One quick network hop just 5 milliseconds away?
Assuming worst case processing semantics (global total order), you would be able to process ~10 million times more customer transactions per unit time with option 1 vs option 2. This is seven orders of magnitude.
In my experience, not a whole lot is truly worst case, but most complicated & important business systems (banking/finance/inventory/logistics/crm/etc.) are pretty close if you don't want to be chasing temporal rabbits around all day.
Nobody else saw the problem here. Establishing a http connection, serializing a JSON document, deseralizing it on the other side, then doing "14 == 15" anyway before going back the way it came. Lighting up a hundred thousand lines of code vs one.
This was done as it iterated over large files, millions of items, millions of requests.
They were genuinely confused why it was slow.
For example, ValidationService as described above could be replaced with OpenPolicyAgent and would scale much better.
That said, I could see someone pointing at OpenPolicyAgent and asking why they didn’t separate distributed updates from the policy spec as two distinct projects, this way you could re-use parts of OPA to distribute config files or feature flags. You can now too, but it requires the extra hop of compiling a function to answer.
Come to think of it, distributed state updating is roughly the same general problem that Kubernetes Operator pattern solves.
Another way of putting it, we have cloud primitives but not enough of them, maybe - or they aren’t well enough explained - to make the choices or restrictions of different architectures a bit more obvious? I especially look forward to a future when we get “design systems” for the programming we do with business logic in the cloud.
I’m over-emphasizing cloud here. I mean code you don’t directly have to maintain and write regardless of where it runs or who wrote it.
That said, depending on the type of validation performed, zero trust systems for example are often built using centralized validation endpoints, so it’s not entirely a bad practice.
These days when I do architecture interviews, I always make sure the problem can be solved distributed or monolithic and when they do distributed, I asked them why and to explain the trade-offs. If they do monolithic, I ask them why not distributed. Etc. For senior engineers, it's such a good indicator of whether or not they've truly worked in distributed systems and understand the human side of the problem.
The big O stuff is great, but the human and organizational costs are easily as important to assess as the computational and memory costs.
Micro-services were promoted for a variety of reasons. One of which was facilitating rolling out projects by consulting companies given the economic incentives for such businesses. Related was the VC driven development model that required rolling out features at greater speed than that of thoughtful development. Another was (remains?) the prevalent lack of competence in designing schemas (required for the basic n-tier system culminating in the database tier).