HNHacker News
TopNewBestAskShowJobs

josevalim

4,367 karma · joined January 12, 2008

submissionscomments
josevalim··on Elixir for Humans Who Know Python
> Django does not generate anything through scaffolding like Phoenix

I was one of the co-authors of Devise, which is an integrated authentication solution for Rails (similar in spirit to Django), and you will easily find people who swear that the code generation solutions are miles better. That’s because eventually they’d want to customize how the framework or library work and then, instead of simply being able to change the code, you need to find the exact hook or configuration to get the behavior you want. Eventually, you end up with hodgepodge of changes that you can only understand when looking at the framework and your code side-by-side.

Perhaps this is one of the topics where there is no “superior” answer, besides personal preferences and past experiences. I totally understand why someone would prefer Django or Devise, but our premise is that eventually you will want to customize it, therefore giving you control upfront is the better way to go about it. However, I would say it is incorrect to say they are for demo purposes in both Phoenix and Django cases, as there is plenty of evidence otherwise.

It is also a matter of framework philosophy. Phoenix aims to give all the necessary foundation for building applications, and then not get in your way, rather than owning all aspects of your application lifecycle. I believe frameworks like Ash (which also runs on Phoenix) take the latter approach.

> Oh views are removed now?

To clarify, Views are not removed. Phoenix is still MVC. The difference is that the templates in your view are defined via `use Phoenix.Template`, rather than `use Phoenix.View`. You can migrate (or not) at your convenience.

josevalim··on Ergo: Erlang/OTP patterns implemented in Golang
Nx is more like Numpy, Explorer is the one that provides data frames (closer to dplyr than pandas though). We provide precompiled assets for both, so hopefully it eases installation. :)
josevalim··on Erlang's not about lightweight processes and message passing
Your comment seems to consider Erlang as a tool to build heterogeneous systems, where you have multiple distinct applications, written in the same or different technologies, talking to each other.

In such cases, I would agree with part of your criticisms because it is indeed the wrong tool for the job. Erlang was not designed to solve this problem: the serialization format is centered around Erlang. The distribution messages reflect the semantics of processes, messaging, monitoring, etc. Inter communication is not the focus. Even on its early days, the distribution was used to provide tolerance against hardware failures by running two identical systems. So I find comparing Kafka and Erlang to be an apples to oranges scenario.

In my opinion, Erlang shines for building homogeneous systems: multiple instances of the same application running in a cluster. Precisely because all I need is Erlang. It comes with a unified programming model for both local and distributed execution. Look at Phoenix uses it to provide features such as distributed pubsub and presence out-of-the-box, features which either require external tools - and additional complexity - or simply do not exist in other platforms. And the beauty in designing such solutions is that you start with the concurrent version and then naturally evolve into making it distributed (if necessary).

I also find the comparison equally misses the mark between restarts/fault-tolerance and Kubernetes. Because, once again, they mostly work at different levels. The classical example is using supervisors to model database connections, something you simply cannot delegate to k8s. But a more recent example comes from working on Nx, which communicates to the GPU. You can stream data in and out of the GPU, but what happens when the code streaming data errors out? You need to develop a synchronization mechanism to make sure the GPU does not get stuck. And what happens if the synchronization mechanism fails? With Erlang I can model and test all of those failure scenarios quite elegantly. Perhaps there are better approaches lurking out there, but it certainly isn't k8s.

When it comes to k8s, they mostly complement each other. Erlang tool for restarting _machines_ is basically non-existent (there is -heart or the stand-by system described by Joe) and k8s addresses that. Erlang doesn't have service discovery, k8s covers that gap. But, even then, there is no assumption you must use Erlang clustering. It is opt-in, you don't have to use it, and in the "worst case" scenario, you can deploy Erlang just as any other language.

josevalim··on Pipe Operator (|>) For JavaScript
Elixir handles errors as you describe, the example does not cover it. The pipeline also assumes a single direction, which pushes error handling to the next step, but there are also constructs (case and with) for handling and composing errors in one go.
josevalim··on Phoenix 1.7 is View-less
I would hope it to be long term but it is impossible to know given how fast the JS landscape can change.

But also keep in mind that Phoenix is not tied to any of these. The esbuild bits are only part of the generated app, which you can stick to or change altogether.

So when we moved to esbuild, we broke zero of the existing apps and no deprecations were emitted. The same would happen if we move away from esbuild.

As an example, I moved most of my apps to esbuild (and I have been very happy with it), but one still uses webpack because it uses the Monaco editor and we rely on certain plugins. So our approach has always been “sane defaults” but not getting in the way if you know better or prefer something else.

josevalim··on Phoenix 1.7 is View-less
That’s one aspect. But I would say the fact that existing Phoenix apps can migrate to v1.7 and use other features such as verified routes without going through a major migration is an important aspect that should not be downplayed, especially when other frameworks may have gone through more than one major version in those 7.5 years and broke existing code more than once.
josevalim··on Phoenix 1.7 is View-less
Phoenix by definition assumes your projects will take longer than an afternoon. :)

If you want a quick and dirty API, using Plug directly may be an option: https://gist.github.com/josevalim/fbfc669aa6087c5b79402b399d...

josevalim··on Phoenix 1.7 is View-less
The integration is still there and it is mentioned in our assets guide: https://hexdocs.pm/phoenix/asset_management.html#third-party...

Regarding import maps, I am still slightly skeptical. First they don’t solve all needs of npm (for example, what if you need to precompile your FE code?). Second, I have feeling that every import map management tool will eventually become a package manager through slowly creep in of requirements. At the same time, there is nothing stopping anyone using it on their Phoenix apps if they so desire.

josevalim··on Phoenix 1.7 is View-less
> - still evolving and releases are breaking (no backward compatibility)

To clarify, this is only true for LiveView which has not reached v1.0 yet. Even then, we have been careful with deprecations while exploring a new programming model with several new ideas.

In fact, Phoenix v1.7 is precisely the consolidation of those ideas. I’d say Phoenix v1.7 would be tagged as v2.0 in most web frameworks, but we kept it as v1.x exactly because it is still backwards compatible (and it has been so for 7.5 years!).

The last steps before LiveView v1.0 are likely forms (which you mentioned) and perhaps a better mechanism to stream with appending/prepending/replacing data.

Once it is out, we hope the remaining cons will be quickly addressed, except the boilerplate bits. The boilerplate is always a trade-off: if we hide too much, then the pieces become more coupled and it is harder to understand how they fit together. So I’d say Phoenix strikes the right balance here and if you have a highly formulaic application, projects like Ash may be to your taste. :)

josevalim··on Phoenix 1.7 is View-less
Each LiveView requires a persistent WebSocket connection to the server. This means it does have a different scaling profile than the usual request/response lifecycle, but the Erlang VM is greatly capable of holding millions of connections at the same time and therefore is a perfect fit for the LiveView model.

In fact, LiveView is built on top of the same Phoenix Channels we used to achieve 1 million connections on a single server. So I would say that LiveView fully leverages the Erlang VM strengths. :)

josevalim··on Bumblebee: GPT2, Stable Diffusion, and More in Elixir
Great summary! I would only add the "Smart cells" bit, which can help you automate and/or learn a variety of tasks through a high-level UI while still being able to drop back to code!
josevalim··on Bumblebee: GPT2, Stable Diffusion, and More in Elixir
And we also have a "Smart cell" for plotting charts too, which can help you get started!
josevalim··on Bumblebee: GPT2, Stable Diffusion, and More in Elixir
You can train models with Axon (the neural network library) but it is not yet fully integrated into Bumblebee. We will start exploring those topics next and hope to provide an equally seamless experience.
josevalim··on Bumblebee: GPT2, Stable Diffusion, and More in Elixir
I have given a presentation on the fundamentals of Livebook at the LIVE Workshop @ SPLASH 2022 just this week. The video is not officially out yet but you can check the pre-recorded version [0] (Google Drive link)

0: https://drive.google.com/file/d/1Mw_NEER4VzA1qhFIq6WH9PYLLbN...

josevalim··on Bumblebee: GPT2, Stable Diffusion, and More in Elixir
Definitely. Running ML/AI nodes close to your users on the edge and collocated with your app can be an exciting combo!
josevalim··on Bumblebee: GPT2, Stable Diffusion, and More in Elixir
Yes, we have an open issue to leverage that in a future release. The issue is that MPS misses many operations, so we will see how challenging it will be.
josevalim··on Bumblebee: GPT2, Stable Diffusion, and More in Elixir
The amount of work varies because there are many different design decisions you can make.

We have been working on this (Nx + Axon + Livebook + Bumblebee) for almost 2 years. One person full-time (part-time for the first year) and 3 part-time. But we have taken a foundational approach: instead of providing bindings to high-level libraries, we choose to build the whole foundation in Elixir and only leverage 3rd-party libraries at the compiler level. It looks like this:

  |-----------|
  | Bumblebee | pre-defined models
  |-----------|
  |   Axon    | NN library
  |-----------|
  |    Nx     | tensor library
  |-----------|
  | Compilers |
  |-----------|
Everything is done in Elixir, except for the (pluggable) Compilers layer, which can be written in anything. The ones we use by default are from Google/Facebook and are written in C++.

This design gives us agency and resiliency: we have control over how to evolve the whole stack and we can swap between different compilers depending how state of the art evolves. We are more expressive too, as we are not limited by design decisions done in libraries written in a separate language.

But you can skip ahead provide direct bindings to libraries such as torch + torch.nn. This will cut corners but also will officially tie you to a Python library.

However it is important to notice that embedding a neural network model inside your Phoenix app, as done in the video, is only really practical in a language that can fully leverage concurrency and effectively run both CPU/IO workflows in the same OS process. I am not following the progress of concurrent Ruby but my rough understanding is that it is not there. So in practice you would need to deploy your machine learning models to a separate service anyway.

I hope this helps!

josevalim··on Bumblebee: GPT2, Stable Diffusion, and More in Elixir
I have written a bit about potential benefits in the initial announcement of Nx [0] and then on its first release [1].

In a nutshell, there has been trends in Python (such as JAX and Thinc.ai) that argue functional programming can provide better abstractions and more composable building blocks for deep learning libraries. And I believe Elixir, as a functional language with Lisp-style macros, is in an excellent position to exploit that - as seen in "numerical definitions" which compile a subset of Elixir to the CPU/GPU.

I also think the Erlang VM, with its distribution and network capabilities, can provide exciting developments in the realm of federated and distributed learning. We aren't exploring those aspects yet but we are getting closer to having the foundation to do so.

Regarding ML ops, I believe one main advantage is explained in this video announcement. When deploying a ML model with Nx, you can embed the model within your applications: you don't need a 3rd-party service because we batch and route requests from multiple cores and multiple nodes withing Erlang/Elixir. This can be specially beneficial for projects like Nerves [5] and we will see how it evolves in the long term (as we _just_ announced it).

Finally, one of the benefits on starting from scratch after Python has paved the way is that we can learn from its ecosystem and provide a unified experience. You can think of Nx as Numpy+JAX+TFServing all in one place and we hope that doing so streamlines the developer experience. This also means libraries like Scholar [2] (which aims to serve a similar role as SciPy) and Meow [3] (for Genetic Algorithms) get to use the same abstractions and compile to the CPU/GPU. The latter can show an order of magnitude improvement over other currently used frameworks [4].

[0]: https://dashbit.co/blog/nx-numerical-elixir-is-now-publicly-... [1]: https://dashbit.co/blog/elixir-and-machine-learning-nx-v0.1 [2]: https://github.com/elixir-nx/scholar/ [3]: https://github.com/jonatanklosko/meow [4]: https://dl.acm.org/doi/10.1145/3512290.3528753 [5]: https://www.nerves-project.org/

josevalim··on Bumblebee: GPT2, Stable Diffusion, and More in Elixir
The default compiler is based on Google XLA, which is the same as used by JAX, so expect similar numbers to said Python library. :)
josevalim··on Bumblebee: GPT2, Stable Diffusion, and More in Elixir
I think perhaps the bigger issue is that you will be trailing ahead a new path which means fewer resources if you get stuck. BUT you can join the #machine-learning channel of the Erlang Ecosystem Foundation or ask around in the Elixir Forum!
josevalim··on Bumblebee: GPT2, Stable Diffusion, and More in Elixir
Yes, we have been eyeing IREE but no work has started yet. However, our compiler architecture was designed exactly to make this possible, so it will be definitely welcome if someone decides to trail ahead on this road!
josevalim··on Bumblebee: GPT2, Stable Diffusion, and More in Elixir
Hi everyone, glad to be back here with the official announcement!

It is late here but feel free to drop questions and I will answer them when I am up. Meanwhile I hope you will enjoy the content we put out: the announcements, example apps, and sample notebooks!

josevalim··on Elixir – HUGE Release Coming Soon
Those are all good points.

When we talk about macros and function parameters, however, we want to avoid spreading multiple constants across multiple modules, because they lead to compile-time dependencies (and potentially long compilation cycles), so my go-to approach has been to define a single module with all of my constants (they could be macros or not, it changes per project).

I wonder if this could be a matter of promoting certain styles rather than adding language features?

josevalim··on Elixir – HUGE Release Coming Soon
The main question of the quote is: if we introduce a system that rejects a "large" amount of Elixir code, will developers adopt it? If some language features are only available for "untyped" code, will developers often forego types only to get those features? And if that happens, could we end-up "forking" the Elixir community, where part of the community uses some idioms and the other part uses others (which is a common complaint about large languages)?

The goal is not to reject types but rather to find a theory that mirrors the language as close as possible. I believe balancing these trade-offs will be essential to the adoption and success of a type system in Elixir.

josevalim··on Elixir – HUGE Release Coming Soon
Zero-arity functions are your externally visible module constants. :)
josevalim··on Elixir – HUGE Release Coming Soon
Hi everyone! I want to align everyone expectations and say this is not related to the Elixir language itself but rather the Elixir ecosystem.

The language has been stable for years and non-minor features always discussed upfront through several proposals. With that said, if you are interested on the work happening on types, I wrote recently about it here: https://elixir-lang.org/blog/2022/10/05/my-future-with-elixi...

This announcement is about a new project, which we are very excited about and working on for months. We hope to officially release it tomorrow (Thu UTC). We are dotting the "i"s and crossing the "t"s right now.

I was definitely not expecting this to make to the front page, I should perhaps be more careful the next time. :)

josevalim··on An animated introduction to Elixir
I agree this is a pain and we have an open issue in ExDoc (#1551) to tackle this.

If anyone is interested in submitting a PR, it will be very appreciated!

josevalim··on An animated introduction to Elixir
Elixir has been backwards compatible since v1.0, released more than 7 years ago. Phoenix has also been API stable for a similar period of time and Ecto for 4+ years.

LiveView has changed over time but that’s precisely why it is on a 0.x release cycle. Even then, we do our best to deprecate first whenever possible. But the goal is that established projects have been stable.

In any case, LiveView is nearing 1.0 now, which makes us comfortable enough to promote it as part of the default Phoenix experience.

josevalim··on An animated introduction to Elixir
Awesome! Btw, I sent a PR to Erlang/OTP so it raises a better error message when using an old Erlang version. :)
josevalim··on An animated introduction to Elixir
This is unexpected. Elixir should just pick whatever Erlang is on the path.

This may be a bit too much, but maybe try removing all Erlang and Elixir versions and just picking the latest for both?

If you are on WSL, then `asdf` or similar can do the trick.

← PreviousPage 8 of 15Next →