Or is good speech recognition just that intensive, such that common consumer devices really can't handle the load?
For just speech recognition you can use Kaldi. Its not hard to deploy on your own server. I've been meaning to package it for Ubuntu phones but haven't gotten around to it yet.
Pretty good speech recognition is doable on current mobile hardware. The trained models are large and there is still work ahead shifting the inference to mobile GPUs but it's doable.
It also helps if it's integrated into the OS, if you want it to work invisibly with multiple apps.
Google Now for instance can recognize me saying "Set a timer" or "Turn on WiFi" even while airplane mode is enabled, but start adding in context (such as "Who is Bob Marley?" "When was he born?") and it probably wouldn't work.
Snips organizes all your information and make it searchable intelligently and actionable fast (it predicts your recurrent patterns so you can order a Uber in one tap)