Think about it: if 1% of people are using Bitcoin then that means there's only a 1 in 100 chance that any given transaction you make can be made with Bitcoins. That's an admittedly low degree of usefulness. If you can get 2% of the people to start using Bitcoin it just became twice as useful and half as many transactions will now require you to convert funds back to traditional currencies. At some "critical mass" adoption level, Bitcoin becomes just as useful as any other currency without such a constant need to move in or out of traditional currencies.
It's called the Network Effect and it's a hell of a market force.
Also your argument is invalid because you could say the exact same thing about holding a bank balance in USD while living in Europe: "But I have to exchange these dollars for Euros in order to even use them at all, man this U.S. Dollar is just completely worthless and unnecessarily complex because no one around me will accept it."