The observable universe would have to be a conservative system for Poincaré recurrence to apply to it. I don't think that's possible; isn't it leaking matter that looks like it won't come back?
I don't understand all the properties a system must have so the Poincaré recurrence theorem applies, but as long as we don't know whether the (entire) universe is finite, that's sufficient to show that Poincaré recurrence may be impossible. In that situation, we can either find information that makes it impossible for sure, or otherwise we know that we don't know.
‘Nothing else can affect us‘ doesn't contradict that, because ‘us‘ would cease to exist at the Poincaré recurrence time (if not earlier).
N.B. I'm just pointing out some basic things, while having no clue how much merit the idea has in the first place, even if the universe has finite state space and infinite time. (I mean just the idea of trying to apply the Poincaré recurrence theorem to the whole universe, not Penrose's work.) AFAICT the proof only says that almost every state recurs infinitely often. So if you picked some state at random as your starting state, it's essentially guaranteed that it will recur, but when it comes to applying this to the universe, isn't it presumptuous to regard the big bang as such a state?