Have you considered using a more traditional HMM style recognizer? The stock Kaldi chain model recipe should get you more like 5 or 4.5% WER on LibriSpeech.
We actually started our experiments with Kaldi and even built a dataset out of our files to train on. But we found that Kaldi required a lot of data-prep and a long lead time. Our internal dataset is quite large and data prep is quite easy compared to Kaldi.
You mean you meant to type 91.3% but accidently typed 95%? Were you using your own transaction software?