Ordered a nice far field microphone and raspberry 4. Gonna try to build my own voice assistant with Mycroft.
I had been intending to do something similar to you, but using Snips[2] since it supposedly has on-device voice recognition (and use of Rust, which I like).
1. https://mycroft.ai/get-started/ 2. https://docs.snips.ai/getting-started
So not too worried about this aspect.