Ireland and England have something called the Common Travel Area (https://en.wikipedia.org/wiki/Common_Travel_Area) which is unrelated to the EU. Basically shouldn't be affected really. Only destabilizing thing that comes to mind would be the Scotland + N. Ireland leaving the UK might make them rethink this.