'It would be a shame if you were to join a union and something terrible happened...'