> Where it gets murky for me is how it interacts with databases. There's mnesia which... I don't understand. Is it embedded?
Mnesia is a application that in my experience runs on some of your Erlang nodes; it provides APIs and also a supervision tree that owns the data. Mnesia is a layer on top of key/value stores; there's some out of the box stores it can use, or there's bindings for LevelDB and Roc0ksDB I guess. When you do a Mnesia read/write/etc, that becomes (more or less) a message sent to an mnesia process, so I'd imagine the LevelDB and RocksDB APIs are only called by a specific mnesia process and there should be no attempt to use the same database by multiple Erlang processes, so you're good from a concurrency point of view.
You can use Mnesia from remote Erlang nodes, it's just sending messages, so easy peasy. However, in my use, we would typically put state management services up on the same nodes storing the data in Mnesia, so we'd send an application message to a process running on the node, which would turn that into mnesia reads/writes as appropriate, and send back a response if appropriate.
If you wanted to use an external database like some sort of SQL, you'd need to find a client and then you could interface with that like in any other language. You'd almost certainly want to put some sort of intermediate connection pool together though, cause most databases aren't thrilled with tons of connections (although there was an article today about going to 1M, so maybe). Our needs fit well with key/value though, so I never looked into that.
> Also how do you test the whole thing? You could test individual processes I guess, but testing the interaction between them all seems fairly impossible.
Everybody has a test environment; some people even have a test environment that isn't production. :)
Testing distributed systems isn't simple. You can test the individual processes, but the interaction between all of them can give rise to emergent behavior which is difficult to test. However an important thing to consider is the importance of testing is related to the cost of failures; if you lower the cost of failures enough, comprehensive testing is much less important. Isolation of processes means that in many cases the rest of the system goes on when one piece fails; of course, if that once piece is in the critical path, that's not true. Hot loading means you can update code without losing established state, which can significantly reduce the update cycle time compared to a system where you need to drain users and restart servers or even longer cycles like in mobile development or even lengthier cycles of traditional boxed software. If you were writing Erlang for boxed software, you'd need to spend a lot more time on testing than in a messaging server.