HNHacker News
TopNewBestAskShowJobs

mononcqc

2,658 karma · joined March 17, 2009

Blog: https://ferd.ca books: https://learnyousomeerlang.com, https://erlang-in-anger.com, https://propertesting.com
submissionscomments
mononcqc··on Stuff Goes Bad: Erlang in Anger (2016)
I've since written a fairly complete blog post only on overload management of asynchronous systems at https://ferd.ca/handling-overload.html -- it hasn't made it in the book, though.
mononcqc··on Stuff Goes Bad: Erlang in Anger (2016)
I have not read any Rails book ever and they could therefore not be the inspiration for this one :)

The true rationale behind the book was to give a better tool to train new coworkers that could help us operate the Erlang routing and logging stacks back when I was still at Heroku, and at the same time to provide similarly useful resources to other people maintaining similar software in the broader community.

Most of the operational aspects of Erlang were a bit of black magic, knowing where to dig through experience, having read the right kind of docs, and everyone would carry little gists of incantations and anonymous functions to run on production servers to help figure out what may be going oddly on there by correctly prodding at the VM.

I wrote the 'recon' library along with Erlang in Anger so that instead of just being black magic & oral tradition, people would have a better set of well-encoded practices to help approach the debugging and optimization of running systems.

mononcqc··on Supporting Hypothesis
I've tried to explain how I tend to come up with properties in my own proptests in http://propertesting.com/book_what_is_a_property.html -- this could prove useful as well.
mononcqc··on Supporting Hypothesis
It depends. One of the places where property-based testing buys you the most benefit is when testing systems that are conceptually simple, but use a complex implementation.

In doing so, you tend to have clearly defined properties that are fairly simple to validate, but a code implementation that may have very intricate internal interactions that cause all sorts of bugs.

See http://htmlpreview.github.io/?https://raw.github.com/strange... for example.

mononcqc··on Elixir School
This is, incidentally, one of the oldest arguments Erlang proponents had about the positives of Erlang's generally unfamiliar syntax: it is different because you should approach it differently, and it helps the programmer do a full context switch when working in Erlang. When Elixir came out, I think a lot of Erlang regulars had similar words of warning about it.

For the most part it hasn't proven to be too critical of a problem from my observer point of view. Mostly, it appears that Ruby tends to be slow enough that even if you write suboptimal "ruby in Elixir" rather than properly working to the VM's strengths, you may still get a speedup over the Ruby baseline you had before.

mononcqc··on Yanni – An artificial neural network for Erlang
It should still be pretty slow compared to just doing the neural net propagation as matrix operations
mononcqc··on How Discord Scaled Elixir to 5M Concurrent Users
you're talking about undoing an architecture that lifts the benefits of an entire platform having done the work for you, in order to pick one that is probably easier to migrate to in the future.

You prefer doing all the work mostly from scratch, upfront and straight away, instead of having to maybe adapt a library later if you actually need to migrate.

I think your cost analysis is off.

mononcqc··on How Discord Scaled Elixir to 5M Concurrent Users
You're acting like the distribution and serialization formats of Erlang are not open or documented.

If you're going for your C++ layer, try https://github.com/saleyn/eixx for example.

mononcqc··on How Discord Scaled Elixir to 5M Concurrent Users
The trick is more commonly used when writing to sockets. A socket owner is required for reading, not for writing.

The trick then is that when you need to write lots of data to a socket to just send a copy of it to the writer so they can dump all their data for cheap, but without changing ownership (which is costly).

Also recently I've gotten http://propertesting.com/ out, you might enjoy it :)

mononcqc··on How Discord Scaled Elixir to 5M Concurrent Users
Aside from macros, pretty much all of these have been covered by build tools for a few years now.
mononcqc··on How Discord Scaled Elixir to 5M Concurrent Users
The true evil approach is to send the socket around, not the message, so that there is no copying required no matter what ;)
mononcqc··on How Discord Scaled Elixir to 5M Concurrent Users
I have never seen a language or framework used at a large scale that did not require digging, understanding, and tweaking of the underlying machinery for its specific use case.

The true failure of the platform happens when you cannot do these tweaks and adaptation, or that their cost is shadowed by having written it in a more appropriate technology at a lower time/cost/effort budget.

mononcqc··on How Discord Scaled Elixir to 5M Concurrent Users
https://github.com/happi/theBeamBook for the VM

http://erlang-in-anger.com/ for introspecting it in production

mononcqc··on Erlang/OTP 20.0
Bit of history on the feature.

The hack was an old thing I had written at a hackathon over half a decade ago. It had little cost to regulars who had installed it at least once before. It relied on DETS and had transient issues with corruption of history files, but nothing major enough for anyone to take notice or get mad about it.

I myself ended up working on other projects once that one was good enough (including the search function with ctrl+r in the shell -- that wasn't there at first, and then things like rebar3 and other side projects), and nobody really ever took over or reimplemented a better shell history around it, nor took the time to port it to OTP.

I had also asked for help in the past for a better implementation that didn't rely on DETS, but not a lot of people were interested or knew how to proceed any better than I did. Eventually I just got tired and reimplemented it with the disk_log module and some log fragmentation and rotation to avoid dropping the whole file every time a limit was reached. It took a few weeks of review to get it in and so far nobody has complained about it.

I think the overall lesson is that the moment you have a quick hack to do the job, regulars get used to them, but forget about the beginner's experience where what you have to do is apply all the hacks at once and dear lord does it ever look janky then.

mononcqc··on Erlang/OTP 20.0
Yes. For such cases, you'll want to instead use a C-Node or a Port program to interface with the C code.
mononcqc··on AFLFast vs. Erlang
Wondering if using erlang:halt() rather than init:stop(0) would be better since halt(0) shuts down the VM instantly while init:stop(0) takes time to take down the entire supervision trees in the VM in proper order.
mononcqc··on Tesla confirms “Autopilot” crash in Montana
Moose or Elks don't 'bounce over the top' on a small vehicle. They crush it, rip the roof off, or go through it. It may depend on antlers, angles, speed, and so on.

I've tried to pick the least graphic images possible:

http://img.radio-canada.ca/2015/08/09/635x357/150809_kr04b_v... from http://ici.radio-canada.ca/regions/saguenay-lac/2015/08/09/0... -- which ended up killing the two occupants of the car.

This one is interesting because you may very well hit the moose facing forwards: http://www.snopes.com/photos/accident/moose.asp

I don't think there is much to be gained in terms of safety. Let's not forget that while you may have large moose, there's a range where they can still be much smaller while growing and still damage smaller cars fairly easily.

mononcqc··on Erlang/OTP 19.0 has been released
I think it's probably more of a consequence of the long long discussions that were had at https://github.com/erlang/otp/pull/960 -- with 15 participants from the OTP team, regular Erlangers, and I'm sure a few Elixir people too.

My idea here is not to remove merit from Elixir in helping the ecosystem, but in pointing out that the Erlang community itself is expanding on these ideas of involvement and a more open design.

mononcqc··on Why Go gets criticized so much
That's the thing though. Errors can be data in plenty of other languages. It's just that it's not the only tool you're given, and there's other things you can do without having to reimplement it all from scratch every time.

Where a lot of people will say "go is opinionated", I keep feeling that Go is just not providing facilities I feel are part of what you should get out of the box (much like a standard library is one) and just shifts the burden to me to reimplement them all correctly. In doing so, you also lose the opportunity to have a well-designed approach that gets rather uniform adoption through the community.

mononcqc··on Why Go gets criticized so much
"Passing up the stack" is fairly minimalistic in an environment where you could be running thousands of goroutines, some (many?) of which may share memory.

If you do it that way, you seem to forego a lot of error encapsulation straight away and quickly find yourself into undefined territory.

mononcqc··on Why Go gets criticized so much
The general principles for fault tolerance require Separation of Concerns vis. Error Encapsulation (make sure that the contagion doesn't spread), Fault Detection (make sure that you know that someone is infected), and Fault Identification (you have ebola!).

Error encapsulation (and this applies equally to modules, components, systems, architectures, organizations) is invariably best done at the lowest level possible, which invariably breaks #3 and #4 (fault detection and identification).

what if you have different classes of exception handling mechanisms available for each of these cases? What about languages that may support exceptions, option types, tagged values, multiple return values, signals, continuations and/or whatever mix of them that exists?

The problem with handling everything right away is that it lacks flexibility to handle things in what could be the optimal manner. "One true form of exception handling", to me, sounds as reductionist of an approach as "one true form of concurrency", "one true programming language paradigm", or whatever.

Treating all error conditions / exceptions with the same mechanism will generally ensure that you pick similar stances on encapsulation vs. detection and identification for all error conditions / exceptions, unless you decide to be extra careful about all of that.

Using multiple mechanisms will allow you to pick, case by case, which one you feel is worth breaking depending on the nature of the fault and what your specific application or system requires.

I feel Go is doing a pretty bad job at this.

mononcqc··on Erlang 19.0 Garbage Collector
You may want to use the procket library: https://github.com/msantos/procket/
mononcqc··on The Zen of Erlang
No. The conference organizers didn't have the time to set a recording setup up.

Maybe if I end up giving the talk again somewhere there could be a recording.

mononcqc··on The Zen of Erlang
That's because I changed the image :)
mononcqc··on The Zen of Erlang
Thanks. ended up going with Cygnus CRS Orb-3, which was a rocket failure, unmanned, and also has high res reusable photos. Text is unchanged, which I believe is fine in this context.
mononcqc··on The Zen of Erlang
I am open to suggestions so that I can modify slides for the (possibly) next times I end up giving a version of this talk.
mononcqc··on The Zen of Erlang
"blow it up" was an example of a thing that "could not make sense" for rocket science as a quote. I also did not know (and currently do not know) of significant rocket explosions or failures that didn't result in the loss of human life, sadly.

Looking at the list here https://en.wikipedia.org/wiki/List_of_spaceflight-related_ac... I'm guessing Soyuz 33, STS-1 and a few others would have worked, but any of those would have brought back similar images, whether the space shuttle image was of a complete one or from the challenger explosion, a failure in rocket science reminds you of any of those you have seen; car crashes and airplane crashes are likely the same.

Then again, it's possible the whole slide is in bad taste. I wanted to convey what the 'let it crash' stuff felt to me the first time I heard it, and Challenger's disaster felt both higher profile and more distant in our collective memory than any random disasters I could have used.

I could probably have avoided discussing the topic entirely, but I hoped that the context around it where I think it would obviously be a bad idea to have 'blow it up' as a rocket science motto would save it. It possibly failed.

mononcqc··on The Zen of Erlang
Chapter 3 of Erlang in Anger (http://www.erlang-in-anger.com/) does mention them among other strategies in handling overload (3.2.2). I tried to put as much concise production experience as I could into that manual. Hopefully it proves helpful!
mononcqc··on The Zen of Erlang
I think a lot of these things are experience-related, or usually cemented within a specific implementation. A lot of people may apply these principles correctly because that's what they find works best, without necessarily bringing it to a conscious level, or to a level of explicitness that makes it easy to teach or use.

Garrett Smith is starting to hit on that with http://www.erlangpatterns.org/ and trying to broadcast that kind of information to the rest of the community, but I'm guessing participation hasn't been strong enough to help (I know I haven't participated enough to that website personally)

mononcqc··on The Zen of Erlang
Author here. This is a challenging one, because it is intimately related to what is acceptable or not to your users.

By default you could say that if the storage mechanism must be up and available and it isn't, then the front-end shouldn't be responsive and it should crash.

You could also say that you want the front-end app to be available if the storage layer is offline. This has two possible consequences:

a) you disconnect the front-end and the back-end so that they do not depend on each other. This can be done either through application strategies (you can define the storage app as 'transient' so it can fail without shutting down the system) or by putting the front-end on a different Erlang node.

The latter means that your dependency on the storage back-end is not as direct as it seems.

b) this is my preferred solution, and it requires you to rework what you think of as 'depends on'. If you expect the storage layer to fail and that you must be able to service the front-end anyway, then the architecture demoed in the presentation needs an asterisk.

The reason for this is that the dependency as described crashes if the database is not available, because the storage subtree acts as a proxy for 'the database'. The OTP structure encodes 'my database is available'.

I can rework that requirement to mean 'the storage layer is up and ready to talk to a database'. This is a huge change because it no longer promise the DB is available, it promises that something whose job it is to talk to the DB is available.

You can then change your interface accordingly. I go into some more detail about this in "It's about the guarantees" http://ferd.ca/it-s-about-the-guarantees.html

In a nutshell, the difference in both initialization and supervision approaches is that in the one described in b), the client's callers make the decision about how much failure they can tolerate, not the client itself. The client making the decisions is what is described in the presentation.

Sadly I could not fit all of that and the compromise of supervision structures within the hour I had allocated for my presentation, so this comment and the side-blog post ought to do (I've also put that material in Erlang in Anger, if you happen to grab that free ebook).

← PreviousPage 3 of 11Next →