Pretrained models are very nice to spin up a system and start using it. You need that because training the model is so hard. But, the pretrained models are by no means a good general model. They are trained with narrow params (like #/size of datasets), so the ability to generalize is very low. It's not uncommon to train a system like that to be 5% WER on Switchboard (think MS/IBM), then have that same system perform at 40% WER on other audio.