Hacker News Reader

I had this case in fine tuned small models (up to 27B) in finance and legal, over bigger models (cost-wise, on-prem for a regulated institution, and better perofrmance because we owned the data pipeline)

mrashad74on I'm making LLMs work on small and old devices

Decent!! Also I love the idea of the live chat, would you mind if I put a reference to it on my repo?

mrashad74on I'm making LLMs work on small and old devices

actually the pi zero is a nice target to play with next, will give a shot and report

mrashad74on I'm making LLMs work on small and old devices

I'm also working on this thread in parallel because it has interesting business aspects, would be happy to see your progress when you release it

mrashad74on I'm making LLMs work on small and old devices

I ran YOLO on an FPGA for my bachlor's project degree project like 7 years ago, but this is even more interesting, and I think we can push bigger models with weight streaming. I still have an FPGA lying around.

mrashad74on I'm making LLMs work on small and old devices

a cluster of vapes running an LLM would be insane. a single vape of these though may run a small LLM like few hundred thousand parameters, worth tinkering with.

mrashad74on I'm making LLMs work on small and old devices

it was a fun name honestly, I see people using the term in different funny contexts on twitter/x so I didn't think much actually. but good point.

mrashad74on I'm making LLMs work on small and old devices

ipad Air I tried doesn't have matmul, but there are tricks you can do to work around it actually.

mrashad74on I'm making LLMs work on small and old devices

I agree, this is why in the further experiments I was putting LFM as the main model not qwen, qwen was nice to prove the point for me, but as I wanted to do more work it became unusable and bad quality, LFM is fine, Gemma didn't perform well but I didn't go far with it, I need to check the others,and I tried MiniCPM but not on this project.

Thanks for the pointer!

mrashad74on I'm making LLMs work on small and old devices

running your own hardware and extracting value out of every piece of hardware is important - one for efficieny, not every application needs smth big - and two for applications that need privacy and similar controls

mrashad74on I'm making LLMs work on small and old devices

let's do it, I'm on twitter @rashaddism

mrashad74on I'm making LLMs work on small and old devices

if you're having some cards already try to explore what people are doing with weight streaming and model compression, I got good results on older cards with far better models than the 16B

and the new thing will be "will it run qwen" :D

mrashad74on I'm making LLMs work on small and old devices

streaming weights of the CD drive should be top 10 blessed engineering hacks I've seen in the past 2 weeks.

mrashad74on I'm making LLMs work on small and old devices

long live TI-84 Plus, the first sentient calculator

mrashad74on I'm making LLMs work on small and old devices

yes I worked with this already in legal, banking and industrial automation. small models with a specified task work well on modest hardware.

mrashad74on I'm making LLMs work on small and old devices

Check LFM as a starting pointer : https://www.liquid.ai/models

mrashad74on I'm making LLMs work on small and old devices

Agreed, and it's a new use case to me because actually in the cases I had the clients were okay with getting GPUs and handling the data mess for us, but none of them requested fine tuning, I believe trying to fine tune will be insanely complex from a regulatory prepsective.

mrashad74on I'm making LLMs work on small and old devices

actually your methodology inspired me to try smth related to small models and harnesses I've been thinking about - and seems you got the main pieces (the idea was trying to see what custom harnesses can extract more accurate reasoning from small models)

mrashad74on I'm making LLMs work on small and old devices

this kind of old processors are smth I want to explore honestly, I don't know if the processor capabilities will allow smth interesting to happen or not, but they're for sure on my target list.

I think realistically smth around 5tok/sec is possible with them for 2B models or so if you provided enough ram for streaming. worth a shot

mrashad74on I'm making LLMs work on small and old devices

the kind of one shots you get with fable 5.5 :D (it was a double typo)