2,658 karma · joined March 17, 2009
The true rationale behind the book was to give a better tool to train new coworkers that could help us operate the Erlang routing and logging stacks back when I was still at Heroku, and at the same time to provide similarly useful resources to other people maintaining similar software in the broader community.
Most of the operational aspects of Erlang were a bit of black magic, knowing where to dig through experience, having read the right kind of docs, and everyone would carry little gists of incantations and anonymous functions to run on production servers to help figure out what may be going oddly on there by correctly prodding at the VM.
I wrote the 'recon' library along with Erlang in Anger so that instead of just being black magic & oral tradition, people would have a better set of well-encoded practices to help approach the debugging and optimization of running systems.
In doing so, you tend to have clearly defined properties that are fairly simple to validate, but a code implementation that may have very intricate internal interactions that cause all sorts of bugs.
See http://htmlpreview.github.io/?https://raw.github.com/strange... for example.
For the most part it hasn't proven to be too critical of a problem from my observer point of view. Mostly, it appears that Ruby tends to be slow enough that even if you write suboptimal "ruby in Elixir" rather than properly working to the VM's strengths, you may still get a speedup over the Ruby baseline you had before.
You prefer doing all the work mostly from scratch, upfront and straight away, instead of having to maybe adapt a library later if you actually need to migrate.
I think your cost analysis is off.
If you're going for your C++ layer, try https://github.com/saleyn/eixx for example.
The trick then is that when you need to write lots of data to a socket to just send a copy of it to the writer so they can dump all their data for cheap, but without changing ownership (which is costly).
Also recently I've gotten http://propertesting.com/ out, you might enjoy it :)
The true failure of the platform happens when you cannot do these tweaks and adaptation, or that their cost is shadowed by having written it in a more appropriate technology at a lower time/cost/effort budget.
http://erlang-in-anger.com/ for introspecting it in production
The hack was an old thing I had written at a hackathon over half a decade ago. It had little cost to regulars who had installed it at least once before. It relied on DETS and had transient issues with corruption of history files, but nothing major enough for anyone to take notice or get mad about it.
I myself ended up working on other projects once that one was good enough (including the search function with ctrl+r in the shell -- that wasn't there at first, and then things like rebar3 and other side projects), and nobody really ever took over or reimplemented a better shell history around it, nor took the time to port it to OTP.
I had also asked for help in the past for a better implementation that didn't rely on DETS, but not a lot of people were interested or knew how to proceed any better than I did. Eventually I just got tired and reimplemented it with the disk_log module and some log fragmentation and rotation to avoid dropping the whole file every time a limit was reached. It took a few weeks of review to get it in and so far nobody has complained about it.
I think the overall lesson is that the moment you have a quick hack to do the job, regulars get used to them, but forget about the beginner's experience where what you have to do is apply all the hacks at once and dear lord does it ever look janky then.
I've tried to pick the least graphic images possible:
http://img.radio-canada.ca/2015/08/09/635x357/150809_kr04b_v... from http://ici.radio-canada.ca/regions/saguenay-lac/2015/08/09/0... -- which ended up killing the two occupants of the car.
This one is interesting because you may very well hit the moose facing forwards: http://www.snopes.com/photos/accident/moose.asp
I don't think there is much to be gained in terms of safety. Let's not forget that while you may have large moose, there's a range where they can still be much smaller while growing and still damage smaller cars fairly easily.
My idea here is not to remove merit from Elixir in helping the ecosystem, but in pointing out that the Erlang community itself is expanding on these ideas of involvement and a more open design.
Where a lot of people will say "go is opinionated", I keep feeling that Go is just not providing facilities I feel are part of what you should get out of the box (much like a standard library is one) and just shifts the burden to me to reimplement them all correctly. In doing so, you also lose the opportunity to have a well-designed approach that gets rather uniform adoption through the community.
If you do it that way, you seem to forego a lot of error encapsulation straight away and quickly find yourself into undefined territory.
Error encapsulation (and this applies equally to modules, components, systems, architectures, organizations) is invariably best done at the lowest level possible, which invariably breaks #3 and #4 (fault detection and identification).
what if you have different classes of exception handling mechanisms available for each of these cases? What about languages that may support exceptions, option types, tagged values, multiple return values, signals, continuations and/or whatever mix of them that exists?
The problem with handling everything right away is that it lacks flexibility to handle things in what could be the optimal manner. "One true form of exception handling", to me, sounds as reductionist of an approach as "one true form of concurrency", "one true programming language paradigm", or whatever.
Treating all error conditions / exceptions with the same mechanism will generally ensure that you pick similar stances on encapsulation vs. detection and identification for all error conditions / exceptions, unless you decide to be extra careful about all of that.
Using multiple mechanisms will allow you to pick, case by case, which one you feel is worth breaking depending on the nature of the fault and what your specific application or system requires.
I feel Go is doing a pretty bad job at this.
Maybe if I end up giving the talk again somewhere there could be a recording.
Looking at the list here https://en.wikipedia.org/wiki/List_of_spaceflight-related_ac... I'm guessing Soyuz 33, STS-1 and a few others would have worked, but any of those would have brought back similar images, whether the space shuttle image was of a complete one or from the challenger explosion, a failure in rocket science reminds you of any of those you have seen; car crashes and airplane crashes are likely the same.
Then again, it's possible the whole slide is in bad taste. I wanted to convey what the 'let it crash' stuff felt to me the first time I heard it, and Challenger's disaster felt both higher profile and more distant in our collective memory than any random disasters I could have used.
I could probably have avoided discussing the topic entirely, but I hoped that the context around it where I think it would obviously be a bad idea to have 'blow it up' as a rocket science motto would save it. It possibly failed.
Garrett Smith is starting to hit on that with http://www.erlangpatterns.org/ and trying to broadcast that kind of information to the rest of the community, but I'm guessing participation hasn't been strong enough to help (I know I haven't participated enough to that website personally)
By default you could say that if the storage mechanism must be up and available and it isn't, then the front-end shouldn't be responsive and it should crash.
You could also say that you want the front-end app to be available if the storage layer is offline. This has two possible consequences:
a) you disconnect the front-end and the back-end so that they do not depend on each other. This can be done either through application strategies (you can define the storage app as 'transient' so it can fail without shutting down the system) or by putting the front-end on a different Erlang node.
The latter means that your dependency on the storage back-end is not as direct as it seems.
b) this is my preferred solution, and it requires you to rework what you think of as 'depends on'. If you expect the storage layer to fail and that you must be able to service the front-end anyway, then the architecture demoed in the presentation needs an asterisk.
The reason for this is that the dependency as described crashes if the database is not available, because the storage subtree acts as a proxy for 'the database'. The OTP structure encodes 'my database is available'.
I can rework that requirement to mean 'the storage layer is up and ready to talk to a database'. This is a huge change because it no longer promise the DB is available, it promises that something whose job it is to talk to the DB is available.
You can then change your interface accordingly. I go into some more detail about this in "It's about the guarantees" http://ferd.ca/it-s-about-the-guarantees.html
In a nutshell, the difference in both initialization and supervision approaches is that in the one described in b), the client's callers make the decision about how much failure they can tolerate, not the client itself. The client making the decisions is what is described in the presentation.
Sadly I could not fit all of that and the compromise of supervision structures within the hour I had allocated for my presentation, so this comment and the side-blog post ought to do (I've also put that material in Erlang in Anger, if you happen to grab that free ebook).