What I actually want to build is multi-backend. It's running off of CMUsphinx right now (which was a ridiculous pain to build, but also taught me a lot about the things that power it), but I want the audio backends to be switchable at the change of some strings in a config file, and of course, the proliferation to more platforms.

The output/initialization/running of julius and cmusphinx are very very similar.

I do think CMU Sphinx is pretty good as an always on listener -- I've had times where it would suddenly die after running for hours (which I tried to debug) -- and times that it would run fine after me playing hours of music non-stop.