What about the people who voted against Brexit but still have to suffer the consequences? There are millions, and millions more who couldn't even vote e.g. because they were too young at the time. "You voted for this so suffer" seems like a remarkably cruel response.
And don't forget people who live here who literally couldn't vote at the time and also have to suffer the consequences.