You could imagine that "you" is simply an artefact of memory. Because it creates a sense of continuity, it is an easy place to create a barrier outside of which is "not you". If you could remember other people's thoughts, how would it affect your perception of yourself?

Just as a neuron acts in concert with other neurons, but does so independently, so do we act in concert with our surroundings. If not for our memory, that creates identity, how are we separate from the rest of the universe?

Though having said that, there are many other explanations which people believe in very strongly. I'm not attempting to challenge those notions. I'm just musing out loud.