HNHacker News
TopNewBestAskShowJobs

mrashad74

295 karma · joined November 12, 2026

submissionscomments
mrashad74··on I'm making LLMs work on small and old devices
We're setting standards already:D
mrashad74··on I'm making LLMs work on small and old devices
No one will run a slopped 600M or even 16B for anything serious today, the whole goal is to experiment, to push both limits of what can be done with the model, and the hardware itself.

It's fun times actually. And I think such experiments and others posted are both showcasing the limits people are pushing.

mrashad74··on I'm making LLMs work on small and old devices
Or maybe AI bees :(
mrashad74··on I'm making LLMs work on small and old devices
following!
mrashad74··on I'm making LLMs work on small and old devices
Is it even still around? :D would be interesting if actually accessible idk
mrashad74··on I'm making LLMs work on small and old devices
speech models are what I'm trying next because imagine a nice portable meeting transcription device that can hold around without internet and for NDA-ed meetings and so (with permission of course)
mrashad74··on I'm making LLMs work on small and old devices
I would actually like to see the Wii one, I don't have one unfortunately so I can't directly try on it but this esp is interesting I would like to replicate and/or improve the results on it, this technique is similar to the streaming i did from flash but probably more efficient, I need to look it up and understand it more. Thanks for sharing it
mrashad74··on I'm making LLMs work on small and old devices
no electrons were hurt in the process of making this :(
mrashad74··on I'm making LLMs work on small and old devices
Thanks for your kind words :) hope to do something impactful one day with it
mrashad74··on I'm making LLMs work on small and old devices
A year ago 1B parameter models couldn't do much, today with the correct harness and/or fine-tuning they can do more or less serious work (not frontier tho).

I believe in a 6-8 months it will get even better and my TV will be happy.

I didn't try fine tuning so small models honestly but looks interesitng to try.

mrashad74··on I'm making LLMs work on small and old devices
This is extremely interesting, do you have a public repo or write up to share or not yet? if not yet please sign me up when you release it, Cheers!
mrashad74··on I'm making LLMs work on small and old devices
not really it's just a mistake of my side replying to differnt comments - I don't not use any agents for interacting online :D
mrashad74··on I'm making LLMs work on small and old devices
well, the whole idea honestly is curiousity about limits of hardware more than actual application today plus the idea that in the future people can run more capable models (in terms of capability @ small parameters) on such devices.

It makes more practical sense today to just go into 4gb GPUs and many people are working on this already.. on my side of things, it's more on the exploration side for now, until we see what good models get released that can use this capability.

There are many people on github and x running decent models especally MoE models on older GPUs and it's doing nice, I'm personally tinkering with this at home.

mrashad74··on I'm making LLMs work on small and old devices
as of now, it's just for novelty and fun, without serious fine tuning on domain data, you can't really do much with the smaller models.
mrashad74··on I'm making LLMs work on small and old devices
most of the problem is actually brute forcing, so I'm using a brute forcing machine and just giving it pointers to reduce the size of search space for tricks, still fun, nothing frontier I know, but it's nice to see what we can do with our little brute forcing machines.
mrashad74··on I'm making LLMs work on small and old devices
well, yes claude is doing that, and I'm directing it and building the experiments and direction only, I didn't hide it at all :)

Everyone is using LLMs now, so it's acceptable to use it for a fun experiment too I believe.

mrashad74··on I'm making LLMs work on small and old devices
no one said it's amazing or novel, it's just a fun experiement I'm not selling this :) just out of curiousity to explore.
mrashad74··on I'm making LLMs work on small and old devices
Raspberry Pis in general are too easy for this task they don't even need modifications, older iphones seem more interesting due to some work on comptability.
mrashad74··on I'm making LLMs work on small and old devices
pretty interesting, I don't own one, but it's possible probably with an emulator to figure this out, thanks!
mrashad74··on I'm making LLMs work on small and old devices
I mostly agree today, but I'm trying to push for the future where smaller models are more capabable, and I believe it will happen + the fine tuned cases I mentioned in other commments.

On other notes : there is no free lunch, you have to sacrifice something for the other

mrashad74··on I'm making LLMs work on small and old devices
I agree with you, and I'm using claude and other models heavily, the idea is not in the soneshot - because actually as you see in the logs I posted on github, it took around 107 + 50-ish experiments to figure out which models to run which and push the speed.

I'm treating it as a search algorithm project, where I develop the methdology and search for pushing the limit of devices and searching in a big space of possible kernels and so for what works for a specific hardware.

so I'm looking at it from a different prespective, it takes a lot of brute forcing, and claude can do this well, the whole game (to me) is which directions you push the model into to reduce the search space. and how that learning from very small and tight devices you can scale these techniques.

But in essence, I don't disagree with your sentiment - the goal is not 0.6B models, these are extremely trivial as you mentioned and as I showed in the experiments, the goal is 16B+ models, and this as far as I've seen, requires smart methods to search for optimizations and customizations both to the model and inference engines.

mrashad74··on I'm making LLMs work on small and old devices
This is an interesting question, so the idea is, based on experience, fine tuned small models can do narrow tasks very well (my experience running models fine-tuned on customer data as low as 1-2B is very promising) also the goal is to find a breakthrough were instead of the line of work (which I respect) that tries to squeeze much bigger models into smaller hardware like strata and maya, I'm working backward to discover/search for custom kernels and modifications till we exhaust the limit of what a smaller machine is capable of, so instead of people doing 64gb machines and old gpus to run a deepseek, I want to see what is the biggest usable qwen small model can run with reasonable speed on smaller devices, and by scaling as we discover more tricks, normal laptops can unlock more compute - for example a qwen 3.8 27B can do pretty serious work when coupled with a specialized harness - but we are yet to figure out what is the smallest machine that can run it with usable output and usable speed, and this can unlock interesting business.
mrashad74··on I'm making LLMs work on small and old devices
this looks very interesting, it can't run a real (read : working on some real tasks) LLM, yet, but it's actually very interesting device.
mrashad74··on I'm making LLMs work on small and old devices
challenge accepted, I have one lying around, will see.
mrashad74··on I'm making LLMs work on small and old devices
challenge accepted.
mrashad74··on I'm making LLMs work on small and old devices
I have a few phones and tablets laying around, will try to play with..
mrashad74··on I'm making LLMs work on small and old devices
Recently I started this open source project at my AI company, where I'm trying to make LLMs working

Now Reached 16B models running on iPad Air 1st gen and xiaomi tv box, both have less than 1GB of usable memory, and reached some decent speeds given the hardware.

I'm open sourcing the effort done, kernels developed and results for everyone to re-produce and push more, also the modified models are on hugging face : https://huggingface.co/hyperionaidev

I want a more challenging device, what do you think?

← PreviousPage 2 of 2