If you have to appeal to rare cases such as traumatic brain injury, then it is indeed 'really hard' and not of much interest theoretically or practically; the mind is what the brain does, so that massive damage can cause changes is exactly what one would have predicted already, just as one would predict that intelligence, memory, happiness, language and other cognitive traits can be changed by massive damage... (Even more massive damage can radically change one's personality through a process called 'death'.) It's not that it's an 'outlier' to be discarded, it's that it's not important if it's that rare.