It's certainly possible with kuberbetes, but you'll be fighting the kuberbetes paradigm. Plus you're checking in to this kuberbetes middleman for part A to understand the availability of part B, for example.
Kuberbetes is highly devex optimized for stateless services that are kinda cattle. Not all use cases fit so neatly into that rubric.
> Suppose you want to have one part that is hosted on a cloud service and another that is on-prem.
Is this hard with kubernetes? Seems pretty simple to add node taints and set up the appropriate VPN to me.
BEAM is not magic. It will be doing a lot of the same stuff that any other scheduler or message queue will do, so I would personally rather choose the more composable solution rather than have to manage BEAM and be forced to use one family of languages only.
I'm thinking if your company is already running kubernetes, adding BEAM on top is probably more effort than adding a message queue (given my limited understanding of BEAM)
Good news is in elixir, though it is not easy to write (still low effort) most of it is pretty straightforward to read so a junior could probably successfully work through and understand a CR.
The other very nice thing about doing it all in the BEAM is that you can make distribution and availability guarantees part of your testing suite very easy, because its even part of you localdev flow. That also means your devs will write that test, instead of of assuming they understand how that kuberbetes operator works (they don't).