OP here - I went with Vosk because it's pretty trivial to lock input down to a set vocabulary (in this case numbers) and from my testing works relatively well on most browsers. (webkit, firefox, etc)
Maybe there's a way to shrink it down given I only need to support a limited vocabulary domain...