I work with people who actively chose to go to conference in Berlin instead of the UK due to visa issues. The passport control is a real issue one cannot just handwave away.
If they emigrated to the EU and want to travel to the UK then I could see that.