Then the CPU switches to running another program that is allowed to access different memory. The CPU encounters a branch and predicts that it will land at 0x00ABCDEF and begins running attacker code briefly. Since it's working on the victim program, the code can access the victim program's data. After several cycles the CPU realizes it made the wrong prediction and NOPs out all of those instructions. However, the malicious code stores its results by shuffling things on the cache, which aren't rolled back because the CPU doesn't consider that to be nominal.
Therefore, the exploit can access memory from other programs. This is my limited understanding of Spectre. Reading the white paper, it must be much more complicated.
Apparently the CPU would normally stop accesses even speculatively when running from the malicious program (except for Meltdown, where it doesn't even stop that).