As you say, when it comes to the brain, there is only a modest relationship between how things feel and what damage is being done to neurons. Neurons themselves do not feel pain, so it is quite possible that you could take drugs that severely damage your brain without feeling their effects. Conversely, headaches and hangovers reflect irritation of the tissue surrounding your brain or of the nerves leading to it. While in some cases a headache can reflect a serious problem that affects or could affect the brain itself (e.g. meningitis), in most cases headaches don't imply damage to the central nervous system.

I am a neuroscientist (in training, at least, in a subdiscipline that's only somewhat related but does occasionally involve drugging monkeys). There is good evidence that frequent use and probably even rare use of MDMA is bad for brains. Neurons don't necessarily die, but axons disappear. It is not clear that rare use of MDMA can cause lasting cognitive impairment in humans.

While DrugMonkey is sometimes a bit of an asshole, I don't find much fault with his assessment of the current state of MDMA research in this post: http://scientopia.org/blogs/drugmonkey/2012/07/25/single-ora.... Note the caveat in the comments: although MDMA is probably neurotoxic, it is still not clear that the neurotoxicity translates to behaviorally meaningful deficits. There exist studies that have shown such deficits, but there also exist studies that have not. Most studies are correlational; I have yet to see a longitudinal study that provides convincing evidence that MDMA itself is a causative factor.