Hacker News Reader

Hacker News

Comment by shujito | Hacker News Reader

Parent Full thread

shujito

perhaps the pi 1st/2nd gen?

I'd thought of the ESP32 devices but their RAM is too small

View on HN

Pi 1 is too easy. My PicoLM [fork] already runs on it.

It also runs on MS-DOS 80386 2MB RAM, so...

And iPhone 1. And Digital UNIX.

Raspberry Pis in general are too easy for this task they don't even need modifications, older iphones seem more interesting due to some work on comptability.

Try an iPod Touch 4th gen? IIRC it's the last model that doesn't support anything higher than iOS 6, so it would be fun to have a local LLM running in a native app that speaks skeuomorphism.

Machines with unified ram work better? whats the bare minimum Ram you need? Maybe those old SGI or Sun machines could have a second life. SGI boxes made the Terminator come to life on the big screen so now their second act could be to help bring the real terminator to life :P

Yes please, my Indy, O2 and Octane2 are all sitting there waiting for some hot Terminator LLM action.

I have a basement full of Pentium 4-Core 2 Quad machines lying around. Has anyone strung together machines of this era to make something work? I always imagined that even if I were able to provide something like 4-8GB of ram per box the token speed would be measured in tok/hr or something.

this kind of old processors are smth I want to explore honestly, I don't know if the processor capabilities will allow smth interesting to happen or not, but they're for sure on my target list.

I think realistically smth around 5tok/sec is possible with them for 2B models or so if you provided enough ram for streaming. worth a shot

the thing that kills them is the complete lack of any kind of matmul instructions. With my PicoLM fork it's possible to "play" GPT-2-AIdungeon on an iBook PowerPC 1.2Ghz with 1GB RAM (scalar instructions, as I said), but it's only for men of the more... patient variety.

Even a Pi 3 is a no-go for any model of consequence. At the very least get a Pi 4 or old ARM phones with NEON.

ipad Air I tried doesn't have matmul, but there are tricks you can do to work around it actually.

I was looking for the lowest claim in this thread, as I managed to get a nano LLM running in the ≈3mb of SRAM of an FPGA with zero DDR in the loop at 20k tok/s, but 2mb on a 80386, great job.

Have you got a write up for it?

I ran YOLO on an FPGA for my bachlor's project degree project like 7 years ago, but this is even more interesting, and I think we can push bigger models with weight streaming. I still have an FPGA lying around.

I haven't written a new blog post for it yet but there's currently a demo live of my FPGA running bonsai 8b on the same FPGA here

https://www.mikeayles.com/blog/on-chip-llm-kv260/

Its right up against the wall of DDR bandwidth, so I'm trying every trick in the book to get the speed up

Decent!! Also I love the idea of the live chat, would you mind if I put a reference to it on my repo?

Absolutely, feel free. I'll try and remember to send you a message when I write up the big-kev and how it was done, but it isn't as crazy as the 20k tok/s SRAM only approach

Reply on news.ycombinator.com