Hacker News Reader

Comment by Touche | Hacker News Reader

I would love to use Elixir, I find the language to be the natural successor to Ruby, given the trends in languages today. However, everything I've read says that it's pretty hard to deploy. That the cloud options are limited (or take away the advantages of Erlang VM like code hot swapping). Any progress on this front?

View on HN

I think one of the difficulties here is that much of the functionality that comes from, for example, Docker + Kubernetes exists as part of the BEAM VM. A lot of the development being done today in deployment is around Docker and Docker clustering technology, which means that it often feels like you are doing things the old fashioned way when using BEAM clustering & networking technology.

There are ways to set up these systems in symbiotic configurations, like having an Elixir/Erlang/BEAM cluster that has auto-joining thanks to external service discovery. However, to use the hot-swap functionality, you are going to be working with the BEAM VM directly, which means going through the Docker container boundary.

Know that if you choose to go with BEAM and no Docker, you will still have access to amazing deployment features (like hot swapping, as you mentioned), but also clustering, service discovery, monitoring, etc. You're just going to be spending a lot more time at the command line without the pretty interfaces and the sleek modern tools.

To get a better understanding of running BEAM, I highly recommend "Designing for Scalability with Erlang/OTP". You'll have to learn Erlang to get through it, but IMO it's the best introduction.

> you will still have access to amazing deployment features (like hot swapping, as you mentioned), but also clustering, service discovery, monitoring, etc.

Can you elaborate on this statement? My understanding is that you can have these things (minus hot swapping) cleanly on Docker.

Yes, you definitely do. What I meant is if you didn't use Docker and just used BEAM, you would still have access to all those features that are present in Docker + clustering technology.

Thanks, you're definitely right!

I've been running an Elixir app in production on about 20 nodes for 18 months. We just build Debian packages and deploy using Salt.

Additionally, most of the cluster nodes run on Spot Instances in AWS so they are relatively inexpensive also. When a new instance comes up it connects to the cluster and starts serving requests. When an instance is killed, traffic is routed to the remaining nodes. Works great.

Do you have a blog post or anything that explains this process? How many nodes in the cluster? Why did you build Debian packages? How do nodes join/leave the cluster (DNS?)? Do you use ELBs at all?

I really should write a blog post about it :) but right now there isn't one.

Debian packages are easy to work with and we keep them as our primary artifact for any given release.

The number of cluster nodes varies between 10 and 35 depending on the workload at the time.

Nodes join the cluster by contacting one of the three "static" nodes that are not spot instances. If they can talk to at least one of those they get knowledge of the entire cluster. When one of the spot instances is going to be killed there is a notice is the instance metadata. The nodes just watch for that and initiate a normal shutdown if they see it.

Yes we do use ELBs but just as TCP proxies and for TLS termination.

My first ever docker build was a multi-stage elixir compilation. The end result is a docker image that can be deployed to Heroku or k8s that works great. It's not that hard to get a good docker setup for it. It's even easier if you use Mix and not releases (like I did). However, releases are cool.

Code hot swapping is not really docker friendly (obviously). I haven't done that, but work's general philosophy has been to have recoverable processes such that starting up the app will bring up the correct processes loaded with the right data. However, it's definitely possible to setup hot swapping but is going to be a learning curve to it.

Code hot swapping is not particularly friendly at all. It's impressive that you can, but the time required to write the code carefully enough and QA the upgrade doesn't seem to be worth it for most applications.

Sure, a telco switch that has to be up 24x7, hot swapping makes a lot of sense. Most applications can spare the brief downtime to reload the old-fashioned way (or have multiple servers to allow upgrades to be done without downtime).

I'm sorry, I disagree.

Code hot swapping can often trivially be done (by simply compiling and copying the fixed module, then 'l(module)' in the shell) if the module does not change the state record or break function signatures. I have used this countless times to fix a module function with no downtime. You don't always need the full code upgrade procedure, which is indeed far more onerous.

I agree, I should have been more specific. Individual module upgrades are indeed straightforward. That didn't sound like what the poster was describing.

One new service that tries to make Elixir deployment easier is https://gigalixir.com. If Heroku was built with only Elixir in mind, this would be it. Disclaimer: I'm the founder.

This looks great. I'm going to give it a shot. If you'd like feedback I'd be happy to provide it.

Absolutely! I'd love all the feedback I can get. You can leave it here or you can email me directly here (base64 encoded): amVzc2VAZ2lnYWxpeGlyLmNvbQ==

It's not very hard to deploy. Just use the proper tools to compile and build a release (on a VM emulating the same architecture as your server), rsync the resulting files into the server and strat the app.

Getting hot code reload to work is harder, but for a "normal" basic deployment it's just this. App restarts take less than one second of downtime for my app, and that's a cost I'm willing to pay.

You shouldn't do this by hand of course. Write an Ansible playbook or something else that does this for you

"Hard" is relative, of course. For small teams without a release engineer, options like Heroku, now, etc. are much easier than what you are suggesting.

Deploying to Heroku is relatively easy, the only downside is you lose access to clustering (plus hot code upgrades but most people don't use that anyway). There's great documentation available for heroku as well:

https://hexdocs.pm/phoenix/heroku.html

There are already recipes for deploying to Heroku, and I've had no problem using Elixir and Phoenix apps on Heroku for the past 2 years. If you use EngineYard or other bare-metal solutions like AWS or Google Cloud, I would imagine there's something you can glean from that.

Have you ever tried distributed clustering, hot code reloading, remote console or remote observer with Heroku?

Reply on news.ycombinator.com