Hacker News Reader

> A model gets partitioned by layer ranges into stages: layers 0 to 15 on one node, 16 to 31 on the next, and so on down the pipeline.

Numbers in this example are arbitrary. How does it actually work? What if the model’s number of layers is 33, or 34?

Is there a document explaining all constraints of this implementation?

View on HN

It depends on the size / structure of the model… but generally you can check out the MeshLLM huggingface org for pre-computed model splits.

We have a job that will take popular models, and dice them up and post on HF.

When that specific model is served, the layers will be shovelled around the available nodes on the mesh.

Do you have instructions how to dice models myself? Every model in your org is a measly 4bit. I kinda hoped it is as simple as taking a GGUF and throwing at Mesh LLM but I fear it's not going to be that straightforward :)

Reply on news.ycombinator.com