If you don’t understand how something that is a net benefit to the economy improves the lives of everyone who is a part of that economy then I’m not sure I can help you.
But no worries. I can help you: you have seen your house, right ? The university building ? At the front, there's a door. Walk through and don't walk through again until you've talked to 10 young people working there about their lives. Go into the country and talk to anyone. Note how rapidly anecdotal evidence builds up that seems to contradict your basic claim here ?
No Brexit will damage people higher up mostly. That's the view, and that's correct. Or a fact, if you will.
Brexit will take that option away from anyone with lower wages, however.