Does Mycroft need a network connection for voice recognition to work? I wasn't able to conclusively tell from a quick scan of their website but their Get Started document[1] says "Mycroft devices need to be paired with your Mycroft Home account to work", which sounds like it does.
I had been intending to do something similar to you, but using Snips[2] since it supposedly has on-device voice recognition (and use of Rust, which I like).
1. https://mycroft.ai/get-started/ 2. https://docs.snips.ai/getting-started