Application Layering – A Pattern for Extensible Elixir Application Design
aaronrenner.io
aaronrenner.io
One of the interesting side effects is that the whole microservices thing just kinda vaporizes as a consideration when you can just take a chunk of your program and run it on another server or something if you need to. I'm still not doing this as fluidly as I'd like, but I've definitely ripped bits out and also merged some bits before without too much hassle.
One of the major problems I've hit with this is just that the mechanisms for composing these pieces together is all over the board. Do you want to plug in a handler for some chunk of the URL space? That works entirely different from composing in two network services on two different ports, which works entirely differently from creating a new protocol that has plugable components, which works entirely different from composing together two services that both end up sharing the same database connection pool.
The other problem I've had is the availability of code that can be mixed and matched like this. One example I recently fixed up in my code base is that we previously had a module for managing metrics for the entire project, which declared each metric value. It was non-trivial work to take everything that module did and made it so each sub-application could register its own metrics independently. It wasn't hard; the design that leaps to your mind is probably basically what I implemented. But it also added yet another way in which composing these things together is a challenge. Composable configuration is also annoyingly difficult.
There's no one big thing that makes this hard, it's a ton of little things everywhere that afford you tying your program together more tightly than it really has to be. I think that if languages and/or libraries supported this better, it wouldn't be that much more difficult to write solid code that worked this way all the time, but right now there's just impediments everywhere.
I don't think this should be a thing anymore either. Container orchestration is slowly fixing this by default but really, it shouldn't be my server's business to track and figure out where all the other services are. That should be in one place. In production it should be owned by Ops. In local it should be optionally controlled by the dev.
The dev should only have to worry about server names when they're collaborating on a bug fix with the owner of a particular service. The rest of the time they should send it all to a single proxy on a single port, with everybody namespaced. Let the proxy talk to Consul or what have you to sort out the rest.
This is also one of the tricks you use to break up a monolith, but it has other uses as well.
I routinely set my primary web services on one port, and the access to debugging and profiling information is available on a web server running on an entirely different port.
In principle, this is not necessary, because everybody and everything should be able to route by URLs, and of course, everybody is using these tools and is intimately familiar with them. In practice, if I depend on that, those things tend to "fail open" and mean that if I don't explicitly block my debug routes from being exposed, it will tend to end up set up so just anybody will be able to reach them. By having by debug & monitoring on a separate port entirely, it makes it so that functionality tends to fail closed; if I don't open that port to people explicitly, it just stays entirely inaccessible, which is the failure state I'd rather have.
Also, there's a whole world of people who aren't working in an environment where there's a dozen people dedicated to maintaining a kubernetes setup for them, and are still building services that aren't built in that paradigm. Plus it seems to me like having really amazingly separable services, even ones not initially thought of as being separate, is a plus for Kubernetes rather a problem.
I brought up Kubernetes because Kubernetes makes you solve this out of the gate.
As an aside, I find it interesting that we are still contained by the limitations of our surfaces, specifically paper and screens. Hell, one could throw stairs and ladders into that.
The result seems to be a mindset that the holy grail of software excellence is distilling our work down to a tree-based structure that will result in us experiencing some Thanos-like post-snap rest.
Yet, at the risk of mass downvotes, I dare say it doesn't particularly matter in 99% of our work. Unless you are working on software that could take/give life or increase/decrease suffering.
Does that mean we should just throw our hands up and build slop? No, it just means that perhaps we would all be better off if we embraced the ideal that our software is more like an organism than a space shuttle. Birth it, mold it, shape it, feed it. Let it grow into a robust, complex thing. Eventually, it will grow old and you can let it die. It will have been both beautiful and disgusting depending on whom you ask.
The point was to say software is at the very least three dimensional. Reducing it to anything less and applying a handful of labels that all end in "ability" will not have made your life worth living.
</rant>
Go outside today.
I'm sure there was a time I could say exactly what you did but I am not 'normal' with respect to being able to reason about objects. If I help a friend move I am forever re-packing their car to reduce the number of car-loads to one less than they thought they needed.
They try to debunk 'learning styles' routinely but the fact of the matter is that certain media are much more comfortable for many people, and 3-D is bad for certain modes of reasoning. But perhaps as importantly, it's bad for some forms of tooling. For example, "show me the difference between the model now and a year ago when we first discussed this." We use tools like this as a kind of 'isomorphic projection' of something that has way more than 3 dimensions so we can figure out the 'shape' of things.
However, I will admit that as I age, my interest in arbitrary puzzles has declined. I much more enjoy putting together puzzles of scenes or artwork I find attractive.
Funny enough, I would say I used to do just as you describe - rearrange carloads of things or otherwise challenge myself for whatever reason to come in "under budget" on space and time. It's probably from sheer exhaustion that my drive for doing so diminished.
Your closing thoughts inspired me to search for 'isomorphic projection'; that led me to find 'graph isomorphism' [1]. I have not dived deep into that particular page, but the image of Graph H is very interesting as it illustrates what I see in my head when trying to distill software.
https://userdashboard.github.io/home
The software provides UIs for users and administrators and APIs and tests and documentation for accounts, organizations, Stripe subscriptions and Stripe Connect platforms all added through modules. It proxies your application server, and you either serve your application within a template or occupy the entire page, how you write it and what you use is left to you. My software tells you who is using your server, and is fully API-driven in case you want more info.
It's ready to use now, and the core software is ready to go but Connect requires client side JavaScript for France, and subscriptions requires the new payment SCA flows which are in-place but doesn't have tests yet, and localization needs to be finished up and that's all probably another week or so.
I'd love to work on an Elixir codebase the size of the one it sounds like the author is!
Shoot me an email at borski@tinfoilsecurity.com if you’re interested. :)
Edit: Someone downvoted this? There really are some miserable people on HN.
However, I have reserved opinion about the idea that a module should only access its direct children. More than often, I find modules interact with each other forming a loop, either by direct function calls or by message passing. If having to organize all modules into tree structure, I sometimes find it is hard to decide which modules should act as parents without introducing additional layers of abstraction. Also, what if I would like to pass a message from children to their parent? As according to the article, it is better to make children swappable and do not rely on parents' code, either direct calling parental functions or passing message to parent's pid looks bad, all I can think of is to use an event bus, which is extra complexity. Not to mention after so many abstractions module names could be too long to be clear.
Personally now I am good with parent, children, and siblings talking to each other, beyond these is probably too far. Not every module needs to be swappable, and too much abstraction makes stuff harder to understand and not always easier to maintain. Erlang thrived probably even before TDD was there with its "let it crash" philosophy. I do not mind not having 100% test coverage, as long as every crash scenario has been fixed along the way and possibly test guarded.
I think Phoenix contexts are plenty and I’m not sure adding more layers of abstraction is a great idea.