check out https://github.com/mlc-ai/web-llm

IMO the main limitation is access to powerful GPUs for running models locally and the size of some models causing UX problems with cold starts