My father was born in Birmingham, a city in which people of english-descent will be a minority by 2020. When a city is only 40% english-descent, it is no longer an English city.
Karachi is still a pakistani city. Lagos is still a nigerian city. Birmingham is not an english city.
Most brexiters aren't particularly fussed with the finer points of the common market, or EU centralised decision making. They just want border control and feel completely powerless as things stand to change anything. There is no neat referendum question regarding immigration, and there seems to be no way other than a referendum for real democratic hearing of people's views on this issue.