HNHacker News
TopNewBestAskShowJobs

josevalim

4,367 karma · joined January 12, 2008

submissionscomments
josevalim··on Distributed Machine Learning Notebooks with Elixir and Livebook
The current pipeline expects PCM audio blobs and, if data is coming from a microphone in the browser, you can do the initial processing and conversion in the browser (see the JS in this single file Phoenix app speech to text example [0]).

On the other hand, if you expect a variety of formats (mp3, wav, etc), then shelling out or embedding ffmpeg is probably the quickest path to achieve something. The Membrane Framework[1] is an option here too which includes streaming. I believe Lars is going to do a cool demo with Membrane and ML at ElixirConf EU next week.

[0]: https://github.com/elixir-nx/bumblebee/blob/main/examples/ph...

[1]: https://membrane.stream/

josevalim··on Distributed Machine Learning Notebooks with Elixir and Livebook
For distributed training, one important feature is to be able to do GPU-to-GPU communication, such as allreduce, allgather, and all2all. Those are not supported at the moment but they are in our roadmap. At this level, however, it seems the language runtime itself plays a reduced role, so I don't expect the experience to be much different to, say, Python/JAX.

For the second question, my understanding is that all big tech models rely on distributed training, so distributed training is a requisite for competing really.

josevalim··on Distributed Machine Learning Notebooks with Elixir and Livebook
Hi everyone, contributor to Livebook and Nx here.

When we started the Numerical Elixir effort, we were excited about the possibilities of mixing projects like Google XLA's (from Tensorflow) and LibTorch (from PyTorch) with the Erlang VM abilities to run concurrent, distributed, and fault-tolerant software.

I am very glad we are at a point where those ideas are coming to life and I explore part of it in the video. My favorite bit: making the tensor serving implementation cluster distributed took only 400 LOC (including docs and tests!): https://github.com/elixir-nx/nx/pull/1090

I'll be glad to answer questions about Nx or anything from Livebook's launch week!

josevalim··on Elixir: Docker now charges open source orgs $300
The analogy worked fine under the last sentence of your previous comment - now removed - which said something like "assume nothing is given, assume nothing is owned".

Anyway, we can work with your example. Your example is missing the context that Docker images are not always used as an app, but also as a building block.

So let's change the exercise slightly: assume Dashbit owns Hex.pm and posting any package to Hex.pm now requires $300/year or packages will be removed within a month.

As an OSS maintainer, if you do not pay, you will break the deployment of all users of your package. So... would everyone still be fine with it? The cost of running a business and all that?

EDIT: Hex.pm supports custom registries which anyone can self-host. Including built-in tools to help you with that. You can apply this exercise to any open source community with a centralized-by-default package manager with custom registries (which, afaik, are most of them).

josevalim··on Elixir: Docker now charges open source orgs $300
I have replied at the root but I think you are misunderstanding my complaints.

I completely agree that Docker doesn't own me anything. By the same token, I can express my frustration by them vaguely announcing that many Livebook and Elixir deployment pipelines may be broken _within a month_.

Also, it took several years of _free work_ for Elixir to become profitable (and even then, we invest most of the profits right back into the community). And I consider well within my rights to recognize that their open source grant wouldn't have helped me back then and I doubt it will help others now.

Overall, it is not about the cost. It is just that I would rather take my business elsewhere (paid or free).

Here is a quick exercise: imagine I announce the next Elixir versions will require at least $300/year to use. I bet everyone would be fine with it. The cost of running a business and all that, right?

josevalim··on Elixir: Docker now charges open source orgs $300
I was not expecting this to make Hacker News, so before I come out as being entitled (hey, perhaps I am!), I think it is worth adding some context now that I have more characters.

First of all: yes, I agree, it is completely fine if Docker wants to charge for this. As it is completely fine for me to be frustrated with the unclear communication around this (there is a HN discussion on it [1]).

The biggest issue is: I actually won't be affected by this. I don't use images from Docker Hub, the community does. Any large open source community has likely asked, at some point, for Docker Images and they are the ones to be affected. And in a month (or not, the messaging has been unclear), they may find their deployments pipelines broken. Any company that depends on Docker Hub images should be triple checking their Dockerfiles right now because they may need to update them within a month.

Others have mentioned Docker for open source projects [2], but the terms are, honestly, very disappointing. If you have any prospect of making any money, regardless of the amount, then you are automatically rejected. And most open source projects take several years (if ever) to become profitable (for Elixir, that took about 6-7 years).

Again, they don't have to provide an open source grant, but I also don't have to pretend the grant will be useful to most. I can (thankfully) afford the yearly fee but I am drawing a line at paying for distributing the software I write for free (and another line on the extremely limited open source grant application).

Have a good one. :)

[1]: https://news.ycombinator.com/item?id=35154025 [2]: https://www.docker.com/community/open-source/application/

josevalim··on Phoenix 1.7.0
None of your questions are related to the point I have made.

> Do you seriously think Phoenix 1.7 is a more productive and better experience for making average websites in 2023 for an average person?

Pointless question without specifying what is an average person and what is an average website. This will lead nowhere than pointless speculation.

If you want to answer it, go partner with user and developer studies firms, apply proper methodologies and correctly evaluate results.

> By the way, isn't your website elixir-lang in Ruby or using a static generator Jekyll for generating the html pages written in Ruby?

elixir-lang uses Jekyll in Ruby. We wrote the landing page for livebook.dev in JavaScript. And my previous (mostly Ruby) company had a blog in WordPress. Phoenix runs on Phoenix and my new company website runs on Phoenix too.

This question also fully misses the point. There is no one claiming Phoenix (or any other technology) is a superior technology across all users and all use cases. This is literally [a straw man argument][0]. And there is no one claiming Phoenix is useful for static generation.

I would be completely happy with using Elixir if we have well-established static side generators. Other than that, the technology for a static side generator hardly matters. What matters are the contents and you will be surprised to find the Elixir website mostly talks about Elixir.

It is clear there is no attempt in engaging in good faith here, so I am bowing out of future conversations.

0: https://en.wikipedia.org/wiki/Straw_man

josevalim··on Phoenix 1.7.0
I don't think there is a definite answer for "what a property designed framework" is but I can try to explain where Phoenix sits in the possible trade-offs.

One possible approach frameworks use to provide escape hatches is configuration. You ship with a series of defaults and, once you want to change it, you need to find out the proper knob to turn. A downside of this approach is finding the proper knobs when you need to tweak it.

Another approach is code generation: you generate code (or configuration) and keep the knobs clear to user. There are still defaults (and conventions) but the knobs are laid out upfront in your application. The downside here is that having so many knobs upfront may seen daunting or noisy.

Since you mentioned Rails, I will provide references on how Rails and Phoenix use those.

Both frameworks leverage both techniques above, but Phoenix errs more on code generation and Rails more on configuration. Here is a practical example.

Rails has a middleware stack that is part of all applications. This stack is hidden from you. Here is how the generator file for said application looks like:

https://github.com/rails/rails/blob/d0d9e8e576b06c19f8507510...

Phoenix has a similar stack (called plug) and the default stack is part of your application. Here is the generator file for it (it is not part of your app but used to generate it):

https://github.com/phoenixframework/phoenix/blob/3c27a34c27c...

You can look at these approaches and try to compare the pros and cons.

---

My biased opinion: I have worked with both and I prefer the Phoenix approach. I understand someone may find having all steps in your endpoint noisy or daunting, but the plus side is that it takes a glance to see all steps a request goes through and you can tweak it in any way you want.

When comparing to Rails, if you want to insert a middleware in the middle of the default stack, you need to explicitly say before or after which middleware. If you want remove something, you need to state the negation and say "I don't want to have this". Overtime this makes it hard for you to visualize what your application does, because you need to assemble the pieces in your head and use tools to print the stack for you.

This also matters on releasing new framework versions. Because Rails has its own stack, if it changes the default middleware stack in any way, it can slightly change how your code. What if the middleware you were inserting before was moved up? Or removed altogether? Or maybe a middleware you deleted was replaced by another one, with similar functionality. Do you want to remove it too? This can lead to subtle differences of behaviour when upgrading.

The code generation approach requires you to opt-in to the new features, which is, IMO, one of the reasons why Phoenix could avoid breaking changes in the last 8 years or so.

This is in no way a knock on Rails. I am 100% confident the Rails team is aware of those trade-offs and could equally argue for their choices. It also isn't a binary choice either, both frameworks use both approaches, with some general preferences for one over the other.

josevalim··on Phoenix 1.7.0
> It feels so weird to have the templates live there, luckily you can put them wherever you want so it's not a real problem

It is curious how some topics like template collocation are divisive. :) In any case, Phoenix had employed both collocation for LV and non-collocation for controllers, so it makes sense to unify the default experience. But ultimately you are right that it does not care where templates are defined!

josevalim··on Phoenix 1.7.0
Here is more context on streams in case you haven't seen Chris' post: https://fly.io/phoenix-files/phoenix-dev-blog-streams/
josevalim··on Phoenix 1.7.0
> Well maybe the first person to actually rewrite core_components.ex will share it in a Gist or something ;)

Yes, that's pretty much it. And we would also be glad to accept contributions so we can generate different core components out of the box.

josevalim··on Phoenix 1.7.0
Thanks for adding a link. In a nutshell, it should work, if not, please follow up in the forum and folks will be glad to provide guidance!
josevalim··on Phoenix 1.7.0
It is impossible for new projects to be broken with this change. It is a new functionality which existing apps would have to opt-in to use.

I don't understand the need to misrepresent what others have said.

josevalim··on Phoenix 1.7.0
> I don't like this possiblity to use so many syntaxes good

There other side of the coin is having specialized syntax for each construct. For example, PHP currently lists [56 keywords][0], and each keyword generally comes with their own grammar rules (I picked PHP because it was an example you mentioned above).

Elixir, on the other hand, has [15 keywords][1]. Defining modules, functions, conditionals, and so on, all use the same syntax rules. At the end, you end-up with less syntax because the whole language is built on the few affordances above. The other benefit is that developers can also extend the language consistently, since the constructs for designing and extending the language are the same.

> The creator of Elixir himself answered that one of the things he thinks he could do better at the beginning was to make the syntax more strict

If I could go back in time, I would have made it strict since the beginning, but we have already made it strict over time through deprecation warnings in the last 7-8 years. Given a time machine, I'd probably not have changed the notes you mentioned above. :)

0: https://www.php.net/manual/en/reserved.keywords.php 1: https://hexdocs.pm/elixir/syntax-reference.html#reserved-wor...

josevalim··on Elixir for Humans Who Know Python
FWIW, F# also has something similar to `let mut`.
josevalim··on Elixir for Humans Who Know Python
I was not able to write a proposal that was generally accepted by the community. We are still one break-through away from that happening.
josevalim··on Elixir for Humans Who Know Python
> Niche technologies can leave you somewhat trapped in your current job ... and the people negotiating salaries and raises for employers will inevitably exploit this.

I don't think the relationship is that simplistic. On [StackOverflow Survey for 2022][0], the top salaries were for Clojure, Erlang, F#, LISP and Ruby. Elixir comes in 6th and Python comes in 25th. Choosing a niche technology may be a way to increase your salary and have more leverage because you will become a specialist on a niche (an increase of more than 40% when going from Python to Clojure by measure of average salaries).

Of course, it would be reductionist to say "learn Elixir and you will be paid more", but I also think it is inaccurate to say choosing a niche language is a certain trap. The best is to explore the companies, communities, and opportunities around you, especially in relation to which point you are in your career, instead of generalizing pros/cons as cheer-leading or blind optimism.

0: https://survey.stackoverflow.co/2022/#developer-profile

josevalim··on Elixir for Humans Who Know Python
Thanks! FWIW, structure.sql is not needed in Ecto either. You must do that when:

1. You want to start discarding migrations (because you have too many) 2. You had an existing database

Are those never a "trigger" on Django side? I appreciate the opportunity to learn!

josevalim··on Elixir for Humans Who Know Python
Why does it have to be a "major player"? :) Going the way of Scala, Clojure, etc is fine. If you have a community where people are innovating and/or enjoying the journey, companies feel productive but invest and understand the pros and cons of the ecosystem, then to me it is all good.

They estimate there are more than 20 million developers worldwide and, if a technology can reach 1% of that, I personally find that amazing (in the "super made it" kind of way).

I am not saying this in a self-congratulatory way (especially cause we haven't reached it) but just generally. Anyone who worked on any sort of moderately used open source project has likely have gone through several rounds of "who is using it? anyone big?" and I wish people can appreciate the small victories rather than aiming to the highest of goals, which most projects will realistically (and understandably) not reach.

If you or your company needs to reassure the accountant, then those languages are not for you, and there is nothing wrong with that. Other than that, your perspective of companies hiring for Elixir is drastically different than mine. :)

josevalim··on Elixir for Humans Who Know Python
That’s great, thanks for taking the time! Appreciated.
josevalim··on Elixir for Humans Who Know Python
I guess it really depends on your definition of "make it". :)

Elixir today is used by startups, unicorns, fortune 500, and at least two of the FAANG (for whatever it is worth). It is used for web apps, embedded, distributed systems, data processing, and making inroads on AI and machine learning. It has a vibrant community with events around the world, several dozens books, and more.

It is on the [top quadrant of Redmonk][0] and [top 25 on most GitHub language stats][1] while being the second youngest language there (only older than Swift which was created by Apple).

Considering it is a language standing on the shoulder of giants but started by a 30-person company, I would say we definitely made it (I am obviously biased!). Of course there is _a lot_ to improve, realistically speaking it is unlikely we will cross top 10 (maybe that's your definition of "make it") but I also want to be clear that we will be here for quite a while!

  0: https://redmonk.com/sogrady/2022/10/20/language-rankings-6-22/
  1: https://madnight.github.io/githut/#/pull_requests/2022/4
josevalim··on Elixir for Humans Who Know Python
Ack on migrations, tks!

---

For completeness, you can do the same in Ecto:

    from t in MyTable, where: t.key >= 12, select: ["field1", "field2", "field3"]
However, that will still allocate a "MyTable" struct in Ecto. And if that struct is large (say 50 fields), slots for it are allocated (but none of the actual data on the fields).

I am not familiar with Python Object Model enough but, if your Python example still allocates an object with slots for all 47 other fields besides the three selected, then you may still find yourself under circumstances you would rather have a "slim" version of the model and reduce memory allocation. This may be needed when you need to load dozens of thousands of entries into memory for processing or similar.

I am not saying this is a must have but I am just trying to clarify what my initial comment was about. :)

josevalim··on Elixir for Humans Who Know Python
It is totally fine to get spoiled if you consider it is better :D

Your case about structure.sql is interesting. It would be nice if we could automate it somehow but, if we simply tried to re-run a changed migration, then the migration would likely fail because the other operations in it (such as adding fields), you already exist, no?

Do you have any suggestions/ideas on how to tackle this? We could have a "mix ecto.migration.fix" that reverts the last migration, wait until you close/save the file, and migrate again, but I am not sure how useful it would be. How would Django approach this? If you auto-migrate and then do further changes, does it change the existing migration or does it generate new ones?

PS: the changeset functions are pretty much on purpose though. It is important to decouple the validation from the schema because a single schema can have several validations rules (and they can diverge overtime!).

josevalim··on Elixir for Humans Who Know Python
Thanks for the insights, I appreciate it! I will take some time to go through the docs and learn more. One last question for now (I hope): after the migrations are auto-generated, do you check them into version control?

> Models are for developers and generally developers have all the permissions any way

To clarify, the reason why you would want to have read-only models is rather in complex data model cases or for performance reasons. For example, if you have a large model with 50 fields, defining a subset with 10 fields can be beneficial to performance in several queries. Other than that, I agree with you.

josevalim··on Elixir for Humans Who Know Python
If you need to change it, you would add it to the migration, schema, and then in the necessary templates. Some fields are private, read-only, etc. so you would change your templates accordingly. Some fields you want to change via a button press, others via a form, and so on. The whole point is that you can evolve the generated structure based on your needs.

> (i.e you can use views if you want and think you need them)

I was mostly speaking about existing applications. New applications have a clear path forward.

I don’t think this strengthens your argument the way you think. I _bet_ Django also has features that were used in the past and are no longer favored now, but still supported for backwards compatibility. This is in no way a phenomenon exclusive to Phoenix.

josevalim··on Elixir for Humans Who Know Python
If you reuse the variable, then you are not getting any of the alleged benefits of using variables. The same name tells little except it is perhaps the same data type and in Elixir you would also get this information from the module you are invoking:

  text
  |> String.split(“,”)
  |> Enum.join(“ “)
You could also equally add comments between the lines in the dot example:

    df.sort_values(“dep_date”)
      # group and select
      .groupby(…)[…]
Although most of the comments above are discardable (IMO) because they are restating the code.

My point is: sometimes I will break out into variables to get some of the benefits you mention. But forcing all intermediate steps to assign to variables is as harmful as using “|>” or “.” exclusively and forgetting about variables altogether. If you need to add error handling, code comments, etc, you can break out of the pipeline as needed.

josevalim··on Elixir for Humans Who Know Python
If you don’t mind, I have some questions as I would like to learn more. Are you aware of a large models.py for reference and learning purposes? Is it expected to define the schema of all my models in a single place but none of the logic?

Also, don’t you run into scenarios in Django where the automatic migration is not enough and you need to provide custom commands? In such cases, how do you provide them?

And can you have scenarios where you have two models pointing to the same table, but perhaps to a subset of fields (this is specially useful in read-only cases)? How is that handled?

Thank you :)

josevalim··on Elixir for Humans Who Know Python
It is perhaps utopia but it would be nice if we could discuss those topics without falling into camps.

For example, I could argue for hours about the benefits of immutability, but I still believe that imperative loops are clearer than functional ones. There is even a repository with solutions for nested traversals in different languages and the Python one is my preferred by some margin: https://github.com/josevalim/nested-map-reduce-traversal

josevalim··on Elixir for Humans Who Know Python
I would find such principled approach to be unproductive. The same could be said about Python, the `.` is equally syntax sugar for passing the object as first argument. So if we take this Pandas code:

    df.sort_values('dep_date')
      .groupby('name')['duration']
      .transform('cumsum')
If we were to pass df as first argument in nested calls, I would find it less readable. But I would also find using variables in this case to be more noise than helpful:

   sorted_df = df.sort_values('dep_date')
   grouped_duration_df = sorted_df.groupby('name')['duration']
   grouped_duration_df.transform('cumsum')
The variables are just repeating some of the information found on the right side and IMO they end up getting in the way of understanding the whole pipeline.
josevalim··on Elixir for Humans Who Know Python
Please open up an issue.
← PreviousPage 7 of 15Next →