The problem isn't so much that the UK is leaving the EU but that the current government as well as other prominent Brexit figures don't have the foggiest idea how they're going to proceed. They have neither vision nor plan (other than "strong and stable", whatever that's supposed to mean) and that's quite a bit unsettling.
The UK hasn't been conducting its own trade negotiations for decades now. They neither have the time nor the personnel required for having trade agreements in place by the time the UK is supposed to leave the EU if everything goes according to plan. It's not even clear if the UK can - as is commonly stated - easily fall back to WTO rules in case no agreement comes to pass because while the UK is member of the WTO that membership currently is contingent on its EU membership.
So, while Brexit might not spell doomsday for the UK this completely haphazard approach of dealing with it doesn't exactly bode well either.