This is a terrible idea. NLP, with or without LLMs, isn't remotely close to being able to interpret commands in someone's house. And who would want a non-deterministic voice assistant anyway?
Imagine you say, "Alexa, turn up the air conditioning," and Alexa interprets this to mean you want to increase the temperature. There are uncountable and unforeseeable failure modes.
Note that this happens even when talking to humans with near-perfect NLP/brains.
The major flaw with voice assistants in general is that they only make sense if they have a pretty narrow and pre-defined universe of accepted commands, which is incredibly confusing to someone who mistakes them for actual NLP.
It's just a really, really bad UI. I use mine all the time, but only to set timers and turn lights on or off.