They'll still get hit hardest. Weak pound means more expensive imports. UK imports ~40% of it's food [1]. That food will now cost more to import raising prices at the supermarkets. Poorer people spend a larger %'age of their income on essentials like food.
The weak pound means importing oil (priced in USD) will be more expensive. This will affect Petrol prices (we're already seeing a 2-3p rise at the pumps). Poorer people spend a larger %'age of their income on essentials like fuel.
There are a number of knock-on effects from the weak pound. Yes it's good for exports but overall it's going to be a rough couple of years for the already disadvantaged.
Speaking to your point on competition as well - if the UK wishes to join the EU free market they will most likely need to accept freedom of movement. No treaty has ever been agreed with any country without this caveat (that covers the Swiss, Norway etc). It's possible that the UK may be an exception but it's unlikely.