Hacker News Reader

I have a basement full of Pentium 4-Core 2 Quad machines lying around. Has anyone strung together machines of this era to make something work? I always imagined that even if I were able to provide something like 4-8GB of ram per box the token speed would be measured in tok/hr or something.

View on HN

this kind of old processors are smth I want to explore honestly, I don't know if the processor capabilities will allow smth interesting to happen or not, but they're for sure on my target list.

I think realistically smth around 5tok/sec is possible with them for 2B models or so if you provided enough ram for streaming. worth a shot

the thing that kills them is the complete lack of any kind of matmul instructions. With my PicoLM fork it's possible to "play" GPT-2-AIdungeon on an iBook PowerPC 1.2Ghz with 1GB RAM (scalar instructions, as I said), but it's only for men of the more... patient variety.

Even a Pi 3 is a no-go for any model of consequence. At the very least get a Pi 4 or old ARM phones with NEON.

ipad Air I tried doesn't have matmul, but there are tricks you can do to work around it actually.

Reply on news.ycombinator.com