Honestly I think that the reuse of voice actors is even more annoying in Oblivion than the absurd non-conversations NPCs keep having with each other.
It's a lovely redeeming quality of both games, by far. Having the same voice actor for everyone is absolutely freaking hilarious, too, to boot.
I mean, how can you not love this lol: https://youtu.be/N6hVmn9FM7o