There are several flaws in your argument: 1) There is no known polymerase that has a fidelity of 100%. There will always be some number of replication errors when DNA is synthesized. 2) There are lots of mutagens in the environment. UV light, mutagenic chemicals, hell, even radioactive decay of mostly-stable elements will occasionally hit a base and cause a mutation.
So, no, there is certainly no species that meets your criteria for non-evolving.
There are several reasons why genomes remain more stable than others, including - as you stated - replication accuracy (which can nevertheless not be 100%), but also survival chances of random mutations, and environmental factors that make deviations from the plan unfit for survival.
Yet, bacteria in general show tremendous genetic mobility. They're easily the most promiscuous lifeform on the planet, considering the fact they have evolved a P2P genetic data transfer mechanism that doesn't require procreation.
So, all things being equal, the rate of genetic change is expected to vary a lot. It's also worth considering that just because ancient versions are still around, that might still mean there are diverse offspring which are also doing fine. It's not automatically the case that earlier versions get replaced.
> I'd be willing to bet that there is a healthy species of bacteria that has stopped evolving
It sounds weird, but that's mostly a matter of definitions. If a species has stopped evolving, let's say because it developed a 100% accurate replication mechanism, that species will have trouble surviving changing environments. Sooner or later there will be a virus or something else that just wipes it out.