The problem is saying that it is equally likely that the other envelope contains half, or double, the money of the envelope we have chosen.
It is impossible to choose a value from an infinite series (the list of possible monies), with every value having an equal opportunity of being chosen.
Therefore the probability of money distribution must be non-even in some way, and the argument falls to pieces.
It is not reasonable to say "Well, I don't know the probability, so I'll treat it as 50/50".