If one envelope contains 4 times the money of the other envelope, then by swapping you either gain 3x or lose 3x, where x is the amount of money in the smaller envelope. Even in this case, the average return is +3x -3x = 0.