Onboarding to Elixir
underjord.io
underjord.io
https://gist.github.com/ashton314/f0a90a46e39b103635f2be8d8e...
The second one adds a pipe in front, which is super handy. Yours gives me some ideas though…
Ah, figured it out. It's very powerful to be able to call elisp inside snippets. Here's my new Labeled Inspect snippet:
# -*- mode: snippet -*-
# name: Labeled Inspect
# key: lin
# --
IO.inspect($1, label: "$2$1 (`(buffer-name)`:`(line-number-at-pos))`")
# -*- mode: snippet -*-
# name: Piped Inspect
# key: pin
# --
|> IO.inspect(label: "$2$1 (`(buffer-name)`:`(line-number-at-pos))`")I agree that immutable data structures > mutable data structures, and while I don't have experience working with Elixir, it's high on my list of new technologies to play around with when I have time. However, although in theory you can go wild mutating your objects and monkeypatching in Rails, in practice those features have never really been a pain point in any of the Rails applications I've worked on, both in personal and professional contexts. Unless you're a very new programmer who doesn't know any better, you're hopefully not mutating the request object in Rails in hidden places, or monkeypatching Object#to_json (for example).
Rather, the difficult parts have been deciding what logic lives where, and other issues more common to web applications overall than Rails specifically. If anything, Rails' convention-driven philosophy (which it sounds like Phoenix follows as well) makes development significantly easier than it otherwise would be, in my experience.
You have to, that's how rails works.
Rails does an enormous amount of damage right out rails new.
The platform involves no compilation, and if you ship the code without tests (which is more often the case unfortunately), things are going to crap out only in Production. Not just that, development also becomes slow because of increasing codebase size, engineers start to get confused and run in circles when trying to find what came from where. Not to mention the non-existent concurrency model.
Elixir, though with a steep learning curve seems to continue to pique my interest and curiosity.
As a second year college student I did a <3,000 loc toy project to learn Elixir, and then used it (with Phoenix and Phoenix LiveView) for my first job, building a dashboard for a nonprofit. While there are bits I'd definitely rather refactor given infinite budget, overall I'm surprised by how well the project went.
The dashboard has been humming along with zero application-level maintaince (just upgrading the Ubuntu version of the server) for months, despite being built by a novice and integrating with an extremely flaky third party API. It was very easy to set up a scheduled ETL job with Oban and add a status page so that they can see if any import has failed recently.
It's not Phoenix, though, and like 95% of discussion about Elixir seems to assume you're using Phoenix, like how so much writing about Ruby assumes you're in Rails, but to an even stronger degree. So maybe that's the difference?
Thankfully, Jose appreciates performance and engineering more too, traits the Rails community was as keen about.
The rails team itself did a lot of work to improve performance.
Rails core teams even spent considerable time working on MRI to improve performance.
Thousands of libraries devoted to this very topic just for Rails.
I mean, you're not even right.
Jose has really thread a needle with Elixir I feel.
Rails has a strong reputation for productivity, it doesn't have the strongest reputation for long-term maintainability (though it can certainly be maintained). Or that's what I hear from the ex-Ruby devs in Elixir. I came via Python and I'd say the same of Django.
There was kind of something in the zeitgeist ready for this... This was a video by Gary Bernhardt in 2012 (that's like, Elixir -2 years?). I've queued it up to where he talks about basically wanting to go live on a cruise ship, fork rubinius, and rebuild ruby on top of erlang. The main chunk of the talk is about the software pattern that Phoenix and Elixir really tries to get you to adopt.
Either the thing you're scaling / respawning is stateless, and then anything from kubernetes to amazon ecs will do.
or it's stateful with a state that you need to persist somehow, and then frankly the hardest part clearly isn't the way web server scale, but rather how the data storage and retrieval mechanism works (along with handling the potential transactionnal behavior requirements).
What am i missing ?
Why though? Why hate k8s so much that you would choose an unreadable language?
Which goes on to say that there is no silver bullet but there is strength in using Kubernetes, it's kind of a lingua franca.
I disagree, do they solve the same kind of problem? Yes. But the same problems? No.
If part of your application needs to be restarted, why would you restart the whole application?
Kubernetes is not an excuse to avoid application resiliency. Erlang/Elixir is not an excuse to avoid infrastructure resiliency.
Part of your program may crash, your node may crash. Both needs to be restarted.
It should definitely be possible. With protocols and behaviours it should be easy to build plugins that "just work".
In my experience with Elixir, bringing everything into the code reduces the surface area of complexity and makes it much easier to model in your head.
Also... the rhetoric here on HN about Erlang/Elixir vs. Kube really needs to stop. People are misinformed. They are not mutually exclusive, you do not need to choose one or the other. We need to reconcile that because the discourse always goes to this weird war in the comments.
If it's not a random example and if you're at liberty to say, would you mind elaborating further how Elixir specifically excels in that particular use case, and what would an alternative approach, regardless how cumbersome, be?
I wonder if there are other patterns like that.
Being able to tightly define what part of your app is impacted if one of these fail and restore it before the rest of the app is impacted is a game changer.
OTP also deeply force you to think of how you start and how you stop these, taking dependencies into account. It helps in cleaning up stuff.
I guess i'll have to try OTP one time to understand why so many people rave about it. I only know one guy who had made real work with erlang / otp and he told me it was beautiful, but not quite as relevant anymore, because of k8s. But i guess i need to make my own opinion.
With respect to user data, a supervised process can crash and the supervisor can restart with the last known good state of that data. It requires some thought on the part of the developer to achieve this, but it's really powerful.
Plus some 3rd party APIs are rather terrible and if you end up re-sending a request (because your server/container went down after the request got sent to the API, but before the response got back/was processed) then the 3rd party API could end up in an inconsistent state.
[0] https://blog.plataformatec.com.br/2019/10/kubernetes-and-the...
Resiliency of a cluster with k8s or similar orchestration, AND resiliency of the individual nodes within that cluster, taking advantage of supervision trees.
(Also, to complete this picture and provide resiliency of the service as a whole, add load balancing across multiple availability zones.)
Restarting an Erlang process and restarting a Docker container is not the same.
Let's say you create an Erlang process per request and that process crash, you don't want to restart the whole node, do you?
Application resiliency and Infrastructure resiliency are not incompatible, in fact, they are complementary.
Also, use libcluster[1] and horde[2], and you will never deal with leader election again. (which Kubernetes does not provide).
I've seen many Kubernetes Operators not handling more than one replicas, and that's a shame if you want High Availability.
[1] - https://hexdocs.pm/libcluster/readme.html
[2] - https://hexdocs.pm/horde/getting_started.htmlSure it is more powerful and does a lot more things. It is also insanely more complex and hard to use compared to Erlang.
if you are going to rely on a stateful service to do this for you eventually the latencies will add up. for in-memory data, you're going to take a trip to redis and back. For persisted data, you're going to take a trip to postgres and back. Plus internal locks in those systems. Plus if you get bigger, you will need sharding logic, and maybe even another distributed shim (like galera or greenplum) on top, which adds even more latency.
For a real-world example of where supervision trees were very important, I once managed a virtual machine orchestrator I'd written in elixir, and they periodically polled the virtual machines to retrieve state. It's a nontrivial question, what do you do when the state of the system is not what you expect? (perhaps something IRL has changed though a side door) And what do you do if you have multiple concurrent independent systems that are dependent on that real-world state?
With a supervision tree, the solution is simple (1 LOC), you slay the parent supervisor and trigger ALL the relevant processes (and no processes, say, checking in on a different machine) to re-establish the ground truth. And you don't have to wait for containers to spin up, etc, because erlang processes are snappy, so there is less downtime. In the system I was managing there were many, many internal errors (netsplits, etc) and the customers generally didn't notice.
I've retained the principle even as I've left the Erlang ecosystem because it turns out to be a fairly convenient way of structuring a lot of concurrent network problems. Reifying servers as explicit things in your codebase is really helpful; at that point you might as well have them restart and all that jazz, but to me the real virtue is just the utility of the organizational scheme to my code. That they are composable is also pretty useful.
I am literally taking a brief break just now from taking one server that I was writing that was maintaining a connection to an authorization service and translating that into local actions and splitting it into two servers, one that maintains and manages the connection and one that is solely responsible for the logic of dealing with what's coming in and out from that server, turning what was becoming rather nasty intermingled code into two separated bits of code that each make their own sense. Because the server abstraction is composable, the entire rest of the program doesn't notice; previously I had one "service", and now I've replaced that service with a (sub-)supervisor that starts the two services up. The interface to the rest of the program is identical; the same "start" and "stop" operations are done, and the "tree"-ness is automatic. The rest of the program doesn't care that what used to be one running server is now two cooperating servers because the composability of the concept of "server" hides that away cleanly.
With the servers reified, the process of starting the system is also clean and easy to explain to other people, rather than bespoke processes for starting each one. I've got fairly granular services here compared to some people, but it's still pretty easy to end up with 5-10 distinct types of "servers" in my programs providing internal services. It's nice that they don't all have their own code for handling errors, getting restarted, managing shutdowns, etc.
It is not night and day. I don't want to oversell this, either. It's not like you'd be crazy to not be using this as an organization technique. You can certainly make large programs without this technique. But it is quite convenient to reify the concept of a "manageable tree of services" as explicit objects in your code, and then be able to deal with them as reified objects, just like anything else that we've found convenient to reify as explicit objects over the years rather than leaving implicit, with bespoke unique interfaces for dealing with each individual object. It's one of those things where, individually, each individual such concept isn't a huge deal, but one of the reasons why 2020s programming can be more effective than 1980s programming is precisely that we've done this to some many things and reduced the number of bespoke one-offs we have to deal with. Explicit service trees aren't going to change your life forever but it's one more incremental improvement that's worth adding to the stack.
The common idea is: there is a current (possibly wrong) state of the world, and there is a desired state.
Let the system use control feedback to automatically bring itself into a desired state.
i.e.
the current state: I have 3 workers.
the desired state: I want 5 workers.
K8s has a predefined workers topology: deployment -> replica set -> pod -> container. Which uses a small number of patterns (service, StatefullSet, etc.)
While Erlang/OTP allows arbitrary supervision trees composed out of several primitives: one-for-one, one-for-all, etc.
Both have a concept of a cluster composed out of nodes.
K8s does this on the [micro]services level, while Erlang/OTP does it on the clustered monolith level.
K8s has a coarse-grained scheduler, while BEAM has fine-grained.
So genservers are abstracting that,while keeping your code pure
I love that language...
If remote is an option there's really no shortage of people looking for experienced devs from what I've seen. Feel free to reach out if you want some options.
Might be just what you are looking for :)
We don't have that solved in Elixir, I've helped a few find their way into employment and am working with a few on client work right now.
But companies just need to get into the habit of working with less experienced devs. Not specifically an Elixir problem from what I've seen but we certainly have it.
It might be slightly worse due to many early adopters of Elixir being fairly experienced so the pool of devs has tended to be intermediate/senior. This is changing and I expect companies will, eventually, adapt.
The demand for devs all around is going up what I hear. So competition should return with a vengeance.
https://jobs.lever.co/joinpapa/f13afec2-d281-45d7-9789-a8521...
Personally I'm not that keen on type-heavy programming. Do you have good examples of this mix from Typescript, I'd be curious to see what that can be like.
They all have types. Some are just much better-documented than others.
I gather you got my meaning but chose to make your point. But just in case.
Phoenix is ludicrously quick to get from zero to realtime UI and interesting backend work such as polling, consuming updates, sharing state, collaboration.
Having done a bunch of Python I wouldn't discount Django but I would pick Phoenix almost every time at this point. The initial burst of Django doesn't seem like a long-term upside and Elixir/BEAM/Phoenix has a lot of long-term upside in my eyes.
Comparing DjangoPackages[0] and awesome-elixir[1], sure you could say "there's an elixir package for everything", but how battle-tested are they? How well-maintained are they?
If someone is trying to decide between Django and Phoenix, I am not sure how likely it is that the individual would have the necessary Elixir/BEAM experience to be confident in dropping-in erlang libraries.
[0] https://djangopackages.org/ [1] https://github.com/h4cc/awesome-elixir
That said, I delved into elixir and haven't looked back. My ios engineer is always mentioning how crazy fast it is compared to previous stacks he's worked with.
Coming from Node, having to mess with WebSockets in Django feels so backwards, although I do appreciate less cognitive load when you're dealing with more traditional, MVC + REST style apps.
If you can't articulate an answer to that, then I would suggest using whichever your team is most productive in.
I should've asked what are the cases when one outperforms the other? Or, what are the cases where Django is a better choice than Phoenix (besides talent, available libs, etc...).
Or even, better, what are the concrete examples in which one is a better choice--from your own experience?
Given my (fairly superficial) experience with both Django and Phoenix I can see a tonne of similarities: convention over configuration, generators/scaffolding, ORM, fairly similar project structure (to some extent). At the same time things such as realtime/WebSockets feel so unnecessarily complex in Django and so much easier to work with in Node, Elx, Go, literally anything else.
I feel like Django fits in the niche once occupied by RoR, whilst Phoenix has a stronger overlap (in terms of use cases) with something like Next.js + Prisma.
> At the same time things such as realtime/WebSockets feel so unnecessarily complex in Django and so much easier to work with in Node, Elx, Go, literally anything else.
So this is where Elixir shines. If you need highly scalable real-time applications, then it's no contest. If it's a pretty straightforward crud app then it's really going to come down to preference.
> Or even, better, what are the concrete examples in which one is a better choice--from your own experience?
I personally chose Phoenix because of liveview. It removed the need for me to maintain what feels like two separate apps when you have a back-end and then a front-end in React/Vue/whatever.
Django still has a bit more stuff out of the box or from the community. There are various pieces that you can use from scratch. For a lower complexity, simple web app if your team hasn't built their "usual needed bits" Django is probably going to give you more.
Elixir/Phoenix is a much better platform that is going to help you more once you're out of the initial bring-up. E.g. they provide authentication templates but explicitly don't just hand you auth because they philosophically believe that it's too integrated with your app that "one size fits all" can ever work (I agree with this). Once you need to do backend work, it's just more Elixir rather than Unicorn etc.
My initial thoughts...
- way less out of the box. you’ll have to spend sometime getting a boilerplate up and running, especially if you’re making APIs, as with solutions like pow aren’t the best for that use case
- having your tests run instantly and webpack or node running on the same process is just life changing
- there is no equivalent of something like Django REST framework for Phoenix, which is annoying, but writing Phoenix is quite productive so it’s all pretty trivial
- ecto has a huge learning curve if you’re coming from the Django ORM
- you will miss the admin panel, and solutions out there are really janky. stuff like Kaffy, ExAdmin... they’re awful.
- documentation is really low tier compared to Django, especially when it comes to Phoenix. I was spoiled by how good the Django docs are and with Phoenix it’s been mostly digging through old forum threads for basic stuff. Docs assume a lot of prior knowledge about the elixir ecosystem, making it hard for beginners, unlike rails and Django. Their docs need a lot of work.
- community can be a little elitist or idealistic. Try asking how to get JWT auth running on elixir forums, watch it get shot down in minutes because “jwt is insecure”. they can be very friendly but also very unhelpful, sometimes lacking pragmatism
- unmaintained packages left and right
- being able to send email asynchronously or do websocket stuff without worrying about memory leaks is just glorious. doing that in Django channels, deploying to production was just a cycle of crossing my fingers worrying that the slightest bug would take down the entire runtime. python async absolutely sucks garbage
I can’t understate that last point. I’ve deployed multiple socket apps with Channels to production, and it’s always been a major pain in the ass on the long run. I spent months debugging a memory leak in Django Channels that only happened in production. Turns out an external package, sentry-python, wasn’t playing nice with ASGI. Async Django is painful and slow. I wouldn’t wish it on anyone.
I have more thoughts written down somewhere, but this is off the top of my head.
All in all: it’s been interesting, very fun. There’s tradeoffs to make, especially in productivity and documentation, but I feel they’re all worth it. Today’s apps demand speed, real-time, and concurrency. Django just feels stuck in a time warp from the last decade and Django channels isn’t progressing fast enough (add that async python is painful).
Maintainability and reliability is important for me as a solo maker. I want a resilient web app that can operate without babysitting, I'm a full time student.
Performance matters to me as well. I can't have a memory leak taking down my runtime every hour, or a slow request causing midnight support emails.
Elixir is just pleasantly fast at everything: unit tests are async, the server is concurrent, everything is just blazing..
Realtime is important for me because you don't always need it, but when you do, it's nice to have it. Knowing that Phoenix has a rock solid implementation makes me use it more. It helps me effortlessly make next-gen products.
Elixir and Phoenix come with performance/reliability guarantees, maintainability and new capabilities like flawless real-time that make it all worth it for me. This is my next upgrade to my stack and I’m really happy with it so far.
I agree that the automatic Hex docs are often a little sparse - they often just document the library API and don't give examples.
If you are trying to port websocket code to Elixir/Phoenix I'd recommend the "Real-Time Phoenix" book ( https://pragprog.com/titles/sbsockets/real-time-phoenix/). It has some hard won knowledge in it (I don't know the author, I just enjoyed the book).
If you ever need any help with content in the book, or a sounding board for Phoenix questions, feel free to DM me on the forums. I appreciate your points and generally agree with them, even the not-so-good ones.
Are there any concrete areas of the Phoenix/Elixir docs that you could point to? I'd love to make some doc PR's to make it more understandable, especially to Elixir beginners.
> there is no equivalent of something like Django REST framework for Phoenix, which is annoying, but writing Phoenix is quite productive so it’s all pretty trivial
It's not quite ready yet, but I'd recommend keeping an eye on https://www.ash-elixir.org/ It'll even generate an admin for you: https://www.youtube.com/watch?v=aFMLz3cpQ8c (although I haven't used django so I can't compare it to that, although I hear good things about django's admin)
> community can be a little elitist or idealistic. Try asking how to get JWT auth running on elixir forums, watch it get shot down in minutes because “jwt is insecure”. they can be very friendly but also very unhelpful, sometimes lacking pragmatism
Yeah this makes me sad sometimes.
> Maintainability and reliability is important for me as a solo maker. I want a resilient web app that can operate without babysitting, I'm a full time student.
Great point! Have you written about your web app anywhere? I think solo development of an application is a really interesting space that I'd like to investigate further at some point.
Best practices is also another one. I feel they take too little opinions on how to use the framework, and it's difficult to find a best practices source (think a book like Two scoops of Django). This can be jarring to beginners.
I get that they're trying to be configuration first, but... People who use frameworks like Phoenix are trying to escape the freedom of systems that take too few opinions.
Phoenix strikes me as trying to be Flask but also trying to be Rails. I'm not sure it gets that middle ground right.
On the prior knowledge assumption: the tutorials for Phoenix assume too much Rails knowledge. Where Rails takes a beginner friendly approach with RailsGuides, Phoenix immediately assumes you're moving from Rails and the docs can be confusing for that reason.
The docs are sometimes made exactly for people who use Rails and that's very annoying for everyone else.
2.I'll be keeping an eye on Ash, that looks fantastic. An API framework that feels like Rails or DRF would be a killer app.
As a note, it's also baffling to me that Phoenix is trying to go the Rails route and completely ignoring modern API development. There's zero tutorials, just a generator. I really don't believe in LiveView: moving the UI state to the server is just a recipe for disaster.
I feel if Phoenix doesn't try to move where the industry is heading (API-first), they'll rapidly fall into irrelevance. Rails can afford to take that risk, they're too big to fail.
Phoenix and Elixir can't. They're too small and the incentives to move are not huge. They didn't kill Rails.
4. I haven't written extensively about it, but you can read my journey at my Twitter account (link in profile). I haven't been too active though, taking a break.
> 4. I haven't written extensively about it, but you can read my journey at my Twitter account (link in profile). I haven't been too active though, taking a break.
Cool, just followed you! Although I'm not really very active on Twitter either, I mostly just interact with whatever @elixirlang retweets (which I read via the Elixir Slack).
Phoenix with Absinthe is great for GraphQL-based API-first systems.
With Absinthe GraphQL, you don't need to to use much of the Phoenix's controllers/views/templates. You write GraphQL resolvers instead of Phoenix controllers and they talk to your Phoenix contexts which either using Ecto to communicate to DB, or talk to other services / APIs. GraphQL subscriptions are event simpler, you either auto-trigger them from mutations, or trigger them programmatic from your code using Absinthe.Subscription.publish/3.
IMO it's best to implement Web UI using Phoenix LiveView and mobile/offline/desktop/SPA UIs and publicly-facing APIs using Absinthe GraphQL.
Feel free to join the discord as well if you just want to follow along: https://discord.gg/WZCECgMe
This thing looks mind-blowing, and could be the future of the Elixir ecosystem. Keep it up!
Pure functional programming makes it easy for beginners to avoid mistakes but it limits their growth and prevents then from learning about proper blackboxing and separation of concerns.
You cannot learn blackboxing without co-locating logic and state. It is an essential coding skill.
When I said that I've never seen any well written Elixir or pure FP project in my career and challenged people to send me a link to such a project on GitHub, nobody could do this.
All I've ever seen in pure FP is spaghetti code; full of dependency injections which make it difficult to follow the logic (where does that module come from?), global state store (no different from using global variables!) or state being passed around all over the code (traversing many files since components are not allowed to hold any state), etc...
Some ideas of FP are useful, but pure FP as a philosophy is rubbish.
You've also extrapolated your anecdotes to make a sweeping statement that implies that all FP is spaghetti.
You've made no objective counterpoints. Reducing any platform or language to "bad" sounds like someone that never quite understood the benefits of FP. Imperative languages are no silver bullet. Literally every code base that I'd ever worked in exhibits the same issues you describe as "spaghetti". But how does FP make it more so?
Do you write unit tests? Tell me how you've not come across a codebase in a non-FP language that didn't require some insane fixture or factory or set-up code and it was insanely difficult to reason about because you had to invoke methods in a particular order to set an object up to be tested for a particular scenario. This is not the case with functional languages. You don't get the rug pulled out from under you very often and if so, it's typically centralized in the "actor" or process whose state can change whereas state is EVERYWHERE in most imperative languages and even challenging at times to introspect, especially when encapsulation is done right in some languages.
Just because you've written more detailed points does not mean that they're valid or good. It just means that you have thoughts.
If you've asked people to send you good projects, then you're already in a mental state that disallows you from accepting a good project. There are many. Don't ask, just search on GitHub -- many projects that don't even come from core devs. It's been much easier for me to reason about and push fixes to these projects compared to Ruby or JavaScript projects.
I mean, we get it. You hate Elixir and you hate FP. You're entitled to that opinion, but please don't tell others to stop using it without having worked with it heavily and an accepting mentality to dig beyond the surface.