330 karma · joined March 11, 2024
I didn’t read a lot about Nx and EXLA in the comments, so I thought I’d share my view:
In my opinion, that’s not a strong argument for Elixir, because pretty much every language has libraries like this. Rust, famously, never stops inventing these libraries (https://arewelearningyet.com/), and to illustrate my point, here are some libraries for niche(-ish) languages:
• Crystal (https://github.com/NeuraLegion/shainet)
• Haskell (https://hackage.haskell.org/package/neural)
• Ruby (https://github.com/unagiootoro/ruby-dnn)
• and many more…
They might be suitable for researchers or projects building networks from scratch, but this is already difficult with a library that plays along. For most people, it’s about fine-tuning, combining, and doing inference on existing models (and weights).
People like to criticize PyTorch, but in fairness, with it I can do useful things with recent models (which matters at the speed of things) pretty much the day they have their open source release (thanks to the transformers library). There exists an equivalent to transformers for Elixir (https://github.com/elixir-nx/bumblebee), but it doesn’t support the top 10 trending models on HuggingFace, which is a problem if you want to use them. Qwen 3, for instance, arguably the most useful open source model family at the moment, isn’t supported at all. And this is the issue all of these libraries face: they are not useful for the majority of people needing libraries like this out there.
Also, PyTorch is really well optimized, and there is a good reason that vLLM (yes, it’s not the fastest but pretty fast, supports almost everything, and is the de facto standard for open source LLM inference) is written in Python and PyTorch. Things like Flash Attention 2 are crucial for LLM inference and still hard to find. Yes, OpenXLA is technically very sophisticated, but I never got it working every time I wanted to use it, and JAX seems more comfortable if you want to use TPUs.
It’s still a very strange development that Python runs the AI revolution. Multi-node training might be a lot of fun with Elixir, but here performance (especially making sure the CPU is not stalling the GPU) is vital, which might make BEAM not an ideal choice.
There is hope though. Recently there have been some interesting developments in the Rust world (https://luminalai.com), and of course no HN comment about training and inference is complete without mentioning TinyGrad. But Elixir simply doesn’t make sense to me (for neural networks). I’m sorry.
The blog article you are referring to uses another method to fine-tune models that many other big platforms like Together AI (and even OpenAI themselves) are already supporting: Supervised Fine Tuning (SFT). We are doing Reinforcement Learning using GRPO instead. SFT has the big caveat that it requires good prompt-completion datasets to work, which are rare/hard to curate for many use cases. For GRPO, you (the programmer) don’t even need to know what the correct answer is as long as you can decide if it’s a good answer (P?NP) at its heart, essentially.
You are right that mDNS is a popular choice, and we even have it in our codebase to be able to debug on macOS.
However, mDNS is a flooding/multicast protocol, and although batman-adv optimizes those, it puts unnecessary strain on a mesh network. That’s why we didn't go with it.
It also gave us the opportunity to try out this other, in our view, more elegant approach to discovery.
Since IBSS is not really useful outside of Linux popular device support, like smartphones, it fell out of the picture pretty quickly (sadly). However, you can connect bridges to our mesh network (again exposing a batman-adv feature), which would give you the ability to connect phones (and of course, printers) to your robots.
[1]: http://grafana.freifunk-muensterland.de/goto/lAqihJDHg?orgId...
Another way would be to somehow prove that you belong.
Security is still a big issue. In the current state, there is no security other than application-layer encryption (QUIC & TLS v1.3). That is fine for pilot projects, but it should not be used for anything sensitive. Maybe we should point this out more clearly in the docs. However, some Wi-Fi chips (not the ones on Raspberry Pi, sadly) also allow setting a password in adhoc (IBSS) mode and 802.11s has native support for encryption. In general is here a problem with lack of adoption of standards by the WiFi chip manufacturers and with Broadcom (the chip on the RP) a lack of support in the Linux kernel driver.
We are planning to implement authentication and encryption in the upcoming release, but this might be a paid feature.
Typically, enterprise networks are encrypted via 802.11x (since a leak of the key wouldn't compromise the whole network), and we might be able to build a decentralised Radius server, but I'm not very fond of that idea.
Ideally, the damage one can do by joining the network unauthorized should be very limited anyway, and authentication and encryption should happen on Layer 5.
Would love feedback / inspiration / suggestions
Abstractions for collaboration are currently in the works, and we hope to release that soon. The work on consensus has already started. Your suggestions seem all very interesting, and we'll definitely consider them. We are also currently in the process of talking to potential users to build handy and approachable abstractions for them.
I saw that [freenet](https://docs.freenet.org/components/contracts.html) went with CRDTs, but I think they made it too complicated. We were thinking about a graph (or wide-column) with an engine similar to Kassandara and a frontend like (or ideally just) SurrealDB.
I remember that iroh moved away from libp2p when they dropped IPFS compatibility and moved to a self-built stack: https://www.iroh.computer/blog/a-new-direction-for-iroh When we got started, the capabilities of iroh didn't really fit our bill, but it seems like it's time to reevaluate that. As a former contributor to rust-libp2p, I never quite got the frustration with libp2p that many people have, Iroh included, especially since many of the described problems seemed fixable, and I would have preferred if they did that instead, and libp2p remains the shared base people build these things on.
I remember Actyx being a rust-libp2p user, but I wasn't aware that they failed. Do you have more info? How and why? It would be great if we could learn from them.
Grammar will be reviewed ;) thank you!
I think the size estimation could also be implemented within the provided abstractions (mainly request-response) but might require you to keep track of neighbors. I think you could implement both algorithms by using our SDKs (none for Go yet).
If you need more control or performance, beyond what we expose through our SDKs, you might need to write a custom libp2p behavior and add it to our daemon. The libp2p part is fairly involved, but I would love to help you with that. Either way I would love to help you out :)
I'm so disappointed that I've never seen your blog before. The stuff you write about is so interesting and actually addresses some issues we are facing. I just sent you an email :)
Thank you! So far, we have tested it with Raspberry Pi 4/5. Jetson boards are on backorder. We have some Intel WiFi chips (since they support some stuff we want), and we will get around to trying them next.
The binaries were also tested on x86 machinery.
In general, I'm not too worried about hardware support since batman-adv is quite widely deployed on a diverse set of hardware and the rest is hardware agnostic.