All that's required to explain the Fermi Paradox is that it happen < 1 in an observable volume of the universe. The volume of the universe in which OoL could happen could be vastly larger, but we could never see most of it. Each extremely rare occurrence of intelligent life would never be able to contact any other, as they'd never be causally connected by signals traveling at the speed of light.