Show HN: IttyBittyAI – Qwen2.5:0.5B running on a 2017 Asus smartphone
The phone now boots into postmarketOS, with a small touch screen UI for picking a Wi-Fi network. Once connected, the web server is started and a chat URL provided (the app sets the device hostname to ittybitty so the standard chat URL will be http://ittybitty.local or just the IP address) and a QR code that will take you to the chat UI is provided.
llama.cpp was compiled on the phone itself (the first full build took over an hour). The model is Qwen2.5-0.5B-Instruct at 4-bit (403 MB).
It can't really do a whole lot other than chat and answer questions using it's training data, which is tiny so not super-intelligent on its own, but I added a few helpers to the web app (a calculator, the date/time, Wikipedia lookups). This project was mostly just to see if it would work at all, so the results were relatively impressive.
What I'm seeing:
- ~4.5 tokens/sec generating (generally faster than you can read the output being generated), ~13 tokens/sec reading the prompt - llama-server uses about 570 MB of RAM with the model loaded - two people can chat at once, users wait their turn - I tried SmolLM2-360M first, it was faster (6 tok/s) but it looped and refused to answer things. The 0.5B model is noticeably better at holding a conversation.
The coolest thing is it's portability. Anywhere you go that has a wifi to connect to you can just turn the phone on, connect it to the wifi, and now that network has a little AI chat bot to talk to. If this old crappy smart phone does this well I can only imagine how newer, better, faster, bigger phones would do.
If anybody has this same smart phone and is interested in doing the same thing just let me know and I can give you the custom OS and llamacpp builds to start from.
Not a whole lot to 'show' since it runs inside my local network but here are some screenshots of the mobile app/server and the web UI: