I believe it is possible that our universe is not orientable, and so there may exist closed paths which, if followed, reverse orientation upon returning to the starting point. If you were right-handed and traversed such a loop, you would now be left-handed relative to everyone else, everyone would agree your heart was on the "right" side of your body etc.

In fact even Atiyah touches on this idea (though without speculating that it might be a property of our universe). In Mind, Matter and Mathematics, he says:

"When I was a student in Cambridge our mathematical society invited a distinguished professor of philosophy, C.D. Broad, to give us an evening lecture. He chose to talk on a problem which had much exercised Kant, the difference between right-handed gloves and left-handed gloves. After the lecture, over dinner, I diffidently suggested to Broad that, since Kant’s time, we mathematicians had a much better understanding of “handedness”, or chirality as scientists call it. We could even envisage a universe in which a left-handed glove could wander around to distant regions and return to fit your right-hand. "