Could Google reasonably put their current voice recognition in a small device?
Software that limits itself to specific keywords can absolutely work well. After all, Google and Alexa do it with their wake words. A voice activated timer could be built fairly easily if it hasn't been done already.
General purpose is a lot harder--and then you need the Internet connection for a lot of the queries anyway so there's no real reason to build in local voice recognition if you then can't really do anything useful with it.
http://stevenhickson.blogspot.com/2013/06/installing-and-upd...
[1] https://github.com/StevenHickson/PiAUISuite/blob/master/Voic...
[2] http://jasperproject.github.io/documentation/installation/#i...
[3] https://cmusphinx.github.io/
[4] https://github.com/julius-speech/julius
Edit: I dug a bit deeper and found this comparison [5] which suggests that Kaldi [6] is the best toolkit to use.