Of course we don't even know whether this sense of self is really purely attached to the brain. That brain is still connected to the rest of your body, and who knows what the body chemistry of the body donor will do to the new brain? Maybe we'll end up with a completely new person. Maybe not all memories will survive the transplant, but even if they do, you'll have a person who has the memories of the brain donor, but will be recognised by everybody as the body donor.
Mind you, if you transplant the entire head and not just the brain, this issue will lean a lot more strongly towards the head donor.