Show HN: Lumabri – What if LLMs worked like Napster? | Hacker News Reader

Show HN: Lumabri – What if LLMs worked like Napster?

github.com

9 pointsvforno10 comments

A while ago I started working on Colibrì to see if it was possible to run huge LLMs on a normal computer. The project grew far beyond what I expected, thanks in large part to the HackerNews community.

That led me to a new question:

What if we stopped thinking about one computer?

This is the idea behind Lumabri.

Instead of requiring a single machine to store and run an entire huge model, Lumabri treats a network of normal computers as a shared pool of resources.

One machine might provide disk space, another compute, another a different part of the model. If a required block or expert isn’t available locally, the system can retrieve or execute it on a peer.

This is particularly interesting for Mixture-of-Experts models. A model can have hundreds of billions of parameters, while only a fraction are activated for each token. Rather than moving huge expert weights over the network, Lumabri can send the small activation to a peer that already has the expert and let it execute it.

The goal is for machines to contribute whatever resources they can afford while using the swarm for the rest.

The idea is very much inspired by peer-to-peer systems: users are the infrastructure.

There are obviously major challenges, especially network latency and security. I’m experimenting with peer verification, SHA-256 verification, signed model state, replica selection, failover, and deterministic execution.

Lumabri is still an early experiment. I don’t have a datacenter or a huge GPU cluster, so I’m building it with the hardware I have and trying to find out whether the idea actually makes sense.

With Colibrì I asked:

Can one normal computer run a huge LLM?

With Lumabri I’m asking:

What if many normal computers could become one huge computer?

Feedback welcome.

Repo: https://github.com/JustVugg/lumabri

Open articleView on HN

I recommend reading some foundational materials on federated learning and distributed inference. The fundamental issues are (1) compute (2) latency and (3) security.

On the compute side, you can't load most individual experts into even high-end consumer-grade cards (e.g. RTX 5090) because the weights are simply too big.

On the latency side, you need to ship the activations and wait for the somewhat underpowered cards to actually do the matrix multiplications, then send the results back. This means each token takes hundreds of milliseconds or even more, which is borderline unusable.

Finally if you're running a P2P network, the only way to verify that peers aren't lying to you is to run duplicate calculations - I don't see how SHA-256 or signed model state help here unless I'm missing something (What are you calculating the hash of? What exactly is signing attesting to?).

This is one of the few places where borrowing ideas from cryptocurrencies actually makes sense - you could imagine a system where peers place bonds and forfeit them if they lie about calculations. You can look at projects like Bittensor and others for how these kind of things are currently implemented.

Thanks for the comment. Extremely clear and detailed. To write this project, I researched a lot about methods and how peers should work compared to a server, as well as security concepts, which certainly have more to add. The goal for this type of project was to move from a single-machine Colibri to multiple-machine Lumabri on a LAN to a large number of machines working together in a Napster-style P2P Lumabri. For latency and other issues, we are studying every type of method that can improve it, and we are also writing and testing other things on our test server. Thank you very much, we will continue to improve.

There needs to be some sort of safeguard to prevent people from abusing this and running queries to do something an owner objects to. I'd feel horrible if I discovered that someone used my AI to figure out how to kill someone. Another problem would be privacy. My highly personal queries would be available to whomever ran that query for me thru logging.

This is not to say it's a bad idea, but it needs some thought toward protecting users or at least acknowledgment and warnings that their information might be available to unintended eyes.

absolutely controls and other things will be part of everything for this type of project. Thanks for the comment and the goal is definitely to improve it more and more.

LLMs are already slow on one machine. Now you are going to add network calls to it?

Actually, no, in this case it would be possible that if other peers in the network give up processing or space you would have a speed that you wouldn't have as a single computer.

Here is some feedback: don't put slop in your readme. It's unreadable.

Adding information for this type of project might make the readme seem overloaded with information, but the goal was to show each test and explain it as best as possible. I'll work on it, thank you very much.

I'm a data scientist, software engineer, etc. I use AI all the time. But when I create something for human consumption, I write it by hand. When someone gives me something written by AI I literally think "I could fire this person and just ask AI myself."

It’s a wonderful world because everyone has their own thoughts on how to handle various situations. I respect your point of view. Thank you very much for your time.

Reply on news.ycombinator.com