295 karma · joined November 12, 2026
It's fun times actually. And I think such experiments and others posted are both showcasing the limits people are pushing.
I believe in a 6-8 months it will get even better and my TV will be happy.
I didn't try fine tuning so small models honestly but looks interesitng to try.
It makes more practical sense today to just go into 4gb GPUs and many people are working on this already.. on my side of things, it's more on the exploration side for now, until we see what good models get released that can use this capability.
There are many people on github and x running decent models especally MoE models on older GPUs and it's doing nice, I'm personally tinkering with this at home.
Everyone is using LLMs now, so it's acceptable to use it for a fun experiment too I believe.
On other notes : there is no free lunch, you have to sacrifice something for the other
I'm treating it as a search algorithm project, where I develop the methdology and search for pushing the limit of devices and searching in a big space of possible kernels and so for what works for a specific hardware.
so I'm looking at it from a different prespective, it takes a lot of brute forcing, and claude can do this well, the whole game (to me) is which directions you push the model into to reduce the search space. and how that learning from very small and tight devices you can scale these techniques.
But in essence, I don't disagree with your sentiment - the goal is not 0.6B models, these are extremely trivial as you mentioned and as I showed in the experiments, the goal is 16B+ models, and this as far as I've seen, requires smart methods to search for optimizations and customizations both to the model and inference engines.
Now Reached 16B models running on iPad Air 1st gen and xiaomi tv box, both have less than 1GB of usable memory, and reached some decent speeds given the hardware.
I'm open sourcing the effort done, kernels developed and results for everyone to re-produce and push more, also the modified models are on hugging face : https://huggingface.co/hyperionaidev
I want a more challenging device, what do you think?