Write Elixir NIFs in Rust
docs.rs
docs.rs
Rustler being a Rust library to write Rust NIFs for Elixir. So I guess you have to compile the NIF first and then use it.
On the other hand Zigler seems to be an Elixir library that you give a string of zig code and it probably offloads compilation to the zig compiler. Since Elixir is compiled to bytecode, if you don't modify it won't get unecessary recompiled I guess so the approach has merit I think.
Btw it's a sigil'ed string, that is the result of a function that takes a string and returns one rather than a string literal
NIFs have a major downside in that it can take down the entire BEAM if they misbehave.
And a huge reason why people use Erlang, is its resiliency from outages.
With Rust memory protection, it gives people more confidence in using NIFs for exactly that reason.
Used it for a few projects and had one of those “How could I have lived before this?” moments.
[1] https://dashbit.co/blog/rustler-precompiled
p.s. Explorer[2], the NIF version of Polars[3] is distributed using Rustler pre-compiled, making Python less and less necessary for the Elixir developer or enthusiast
[2] https://news.livebook.dev/data-wrangling-in-elixir-with-expl...
The particular use case is obsolete now that Heex templates have arrived in Phoenix, but the associated example code might be useful.
Interesting idea, since I was enjoying Phoenix and Elixir. However, I jumped too quick I think into making a Rust NIF and it made the whole thing quite complicated. I was finding chasing down errors in multiple domains (Rust->Elixir->JS and back) to be challenging. I learned a ton on that project, however.
I wonder if it would go smoother now due to advancements with Phoenix and less javascript involved. I think some of the Rust libraries I was using have now advanced more as well.
Say I wanted to implement my own ETS, maybe to add an atomic rename (which ETS really should have). Like ETS, I'd like to have a :named_table, so I don't need to pass a `ref` around everywhere. How/where do I create this global table lookup and how/who manages it?
you might be interested in looking at this
Neural: an ets-like interface to shared terms
Sorry, but I don't.
I linked it as an example of handling your own tables by name vs by ref in a Nif.
For global NIF module data, you're steered towards setting it up in the priv_data pointer passed to those hooks, and retreivable with enif_priv_data, but you can also use static variables or whatever else (it is native code, BEAM can't force you to do anything).
Lifecycle of Erlang Terms is bound to environments, often NIFs get Erlang Terms from the process that called them, and return Erlang Terms to that process... You can't keep references to those in native variables beyond the scope of the function call though; if you need to keep things around, you've got to either copy the values directly into native values, or use erl_make_copy to copy to an environment that you manage. (The erl_nif manpage is probably more clear than me, look for ErlNifEnv)
All that said, I think you're looking for a multi-table atomic rename like in MySQL[1]; so that you can replace a table in a single transaction? If that's all you want to change, you probably should just modify ets, rather than reimplement all the things; I'd be somewhat concerned about adding additional locking, but I don't know what the existing locking looks like.
[1] https://dev.mysql.com/doc/refman/8.0/en/rename-table.html
But probably you would wanna not use resources and pass name of a table as an atom into your C level.
Lifecycle is simple, the same lifecycle is as for long binaries. Each process that saw the reference would increase internal ref count. Once a process has been GC-d and does not have references to the resource in the process heap - the ref count decreases. When nobody references the resource - it is removed.
I love there’s a way to make apps on BEAM faster, I hate that you have to do that with non-native Erlang/Elixir language.
If only the BeamASM could be 5x faster than it is, rustler wouldn’t be needed (and come with all the downsides of NIFs).
Of course everyone wants their language to be faster, but the interop story for Elixir/Erlang is very good, all things considered.
* What are NIFs: https://www.erlang.org/doc/tutorial/nif.html
* Using Rustler in Production: https://discord.com/blog/using-rust-to-scale-elixir-for-11-m...
If you implement a NIF in C and you panic then the erlang VM will crash bringing down everything.
If you do it in rustler then that panic is caught and is raised as an exception on the elixir side, which can be handled (or not) like any other exception.
This allows you safe bindings to existing rust libraries (such as polars in the case of explorer)
If the code does something unsound in an unsafe block then yes, you get to keep both halves like in C. For example if you've decided to unsafely index arbitrarily far into a small array, this will blow up just the same as x[n] in C would. But, why would you do that?
The part of having a SecDevOps as part of the many roles one has to perform, is worrying about this boring stuff.
Hence coding in a safer language by itself isn't a guarantee, care must still be taken.
Still better than C language family anyway.
Remember Rust's Safety Culture, the big C word I mention over and over here and elsewhere that we run into each other? Although the Rust compiler doesn't get to have an opinion about this, Rust's culture does, and Rust says no, the error is in your unsafe block.
Suppose I write a function which takes six 8-bit unsigned integers A, B, C, D, E and F, adds them together and then indexes into an array I own which is 512 entries long. Clearly if you give me most possible values of A through F this is a bounds miss.
If I write this function in safe Rust, it panics when you do that. The bug is in my code, that's where the panic happens.
If I write it in C, it blows up when you do that, but, and here's the crucial part, maybe I say "Idiot, the documentation clearly says in paragraph six see value normality subsection B4, and in B4 I wrote that none of A through F should have values such that when added they sum to 512 or more, thus it's your fault.
If I write it in unsafe Rust then culturally I have two clear choices. One of them is like the C. I write my very extensive documentation and I mark my function unsafe which indicates that callers need to read and understand the documentation to ensure they obey all pre-conditions. Their code will also be unsafe because they can't call my unsafe function without that, reminding them of their responsibility.
The other choice, which is more usual, is to safely encapsulate the feature. I can optionally write extensive documentation, but regardless I must ensure the function cannot cause unsafety even if used by a malicious idiot.
Because of these two options, the problem is always the unsafe code. Maybe it's my unsafe code (I did a bad job either obeying or specifying the preconditions) or maybe it's your unsafe code (you didn't obey my preconditions) but either way it's never the safe code.
Rust's compiler can't promise that but Rust's culture can.
Something shared on Java and .NET land, when discussing JNI, P/Invoke, C++/CLI.
For example, I use it to parse XML files while this can be done in Elixir and there are libraries for it. Being able to use the Rust library serde and quick_xml makes life so much easier.
Another example would be the Elixir library Explorer which is built using the Polars library in Rust for fast dataframes.
Rustler is also pretty easy to use. I never coded Rust before and I had working code very quickly and build is easy as running your normal mix commands.
You can also use the rustler-precompiled library so users of your library dont have to worry about building the library.
It's essential the PyO3 of the Elixir ecosystem, excluding the Python to Rust part .
https://dashbit.co/blog/rustler-precompiled
(it's similar to pre-compiled Python wheels, but it's better developers' side in my opinion)
https://github.com/philss/rustler_precompiled
The users of your library doesn't have to install anything. The library will seem like any Elixir library when they use it.
It safe because when you upload your project to Hex.pm you include a file with all the checksum of the latest version of your library and when you install the library it will use those to download the library from GitHub.
I suppose the only downside of using the same code is it has the same bugs (e.g. I forgot to strip whitespace before doing a length check). So you don't get a belt and braces of having two lots of independently implemented validation
But then the upside is that it has the same bugs, so you only have to fix it once