We already do. MLC-LLM and Llama.cpp have Vulkan/OpenCL/Metal 3 bit implementations. That can run llama 7B (or maybe even 13b?) in 8GB.

TBH devices just need more ram for coherent output though. Llama 13b and 33b are so much "smarter" and more coherent than 7B with 3 bit quant.