Because you still have the same expectations about how music should be structured the second time you listen to it. But the effect wears off after some time. Try playing it 20 times on repeat. It won't do it anymore. But that's mostly temporary, if you come back to it after a while it should work again.