The Sorry State of Convenient IPC (2014)
dev.yorhel.nl
dev.yorhel.nl
To connect to another host you just do:
remote:hopen `:server:port;
The port number is specified with -p on the command line. To send messages: answer:remote "sum x"
or if strings are repulsive to you you can pass in an expression like: answer:remote (`sum;x). `sum here is a symbol which is like a reference to a variable (or function). You can also pass a literal function for the remote host to execute, since they're just data.You can also do asynchronous calls using the negative value of the handle (which is just a file descriptor int, wildly enough): neg[remote] (`incViews;`welcomepage;1)
Completely natural and simple. The only issue that I'm aware of is a 2gb limit for each message. Built-in functions can split your data into blocks to transmit large amounts.
The in-memory representation of data and the network representation are essentially the same. So there's no costly per-byte packing or unpacking going on, simply a memcpy(). In terms of performance and power use (environmental impact) this is a huge win.
It's trivial to expose data to the outside world by creating a web server. Simply define a function called .z.ph (or .z.pp for POST). This one evaluates the URL requested as a Q expression and returns the result as json.
.z.ph:{.j.j value x 0}
Being able to build services as simply as this allows you to begin to think about the microservice as the unit of program composition, rather than classes, functions, or modules. Microservices have their own issues, but that's a topic for a later rant.Most linux server apps I know communicate through the filesystem by leaving a .pid,.lock and/or a .sock somewhere and then setting up a custom channel using a handle to that. It feels kind of hacky, but at least you get to control the crappiness.
Edit: Poettering's talk about kdbus from devconf 2014: https://www.youtube.com/watch?v=HPbQzm_iz_k
Not sure why having non-kernel implementations is a problem. Is the fact that there is no non-kernel implementation of Unix sockets / TCP/IP / filesystems / name-most-other-Linux-kernel-features a problem for you?
BTW I don't especially care about kdbus, but I do hate that there are no good, featureful communication primitives in Linux.
There are some non kernel sockets implementations, but it is less of an issue as every kernel implements them, but kdbus is not implemented anywhere yet. It is also not possible to implement correctly in userspace as the security guarantees rely on a kernel, which is inconvenient. It is unlikely eg FreeBSD will ever implement it.
He considers using message queues and dismisses the idea because "you're still on your own in implementing an RPC mechanism on top of that. And for the purpose of building a simple RPC mechanism, I'm convinced that plain old UNIX sockets or TCP will do just fine.".
In my experience, getting an arbitrary sized message from A to B over a potentially unreliable network is the difficult bit and implementing a basic RPC mechanism is pretty easy. I did a project that required a lot of IPC and ended up using nanomsg (also not mentioned in the article.. it's very similar to zeromq) for [fast] reliable message delivery and wrote my own basic RPC layer on top of this (NanomsgRPC - currently C# only ..). This worked pretty well for me.
A note on nanomsg though: I wouldn't consider it stable enough for production use, but that said, it didn't give me any problems for what I used it for.
It turns out that some of these problems already do have pretty decent solutions.