The only challenge in that case is ram (512, cpu is fine), but I think you should try getting something usable on the raspberry pi zero.

Once that's done it means people(or startups) can get an edge LLM subsystem on their projects for the zero or boards with more ram.