If you want to do away with intermediaries, what Bitcoin needs is a large ecosystem, such that my BTC never needs to be converted to USD to be useful - I want to be able to be paid in BTC, then to be able to exchange that same BTC for food. The food vendor in turn needs to be able to exchange that BTC for, say, a car.
For that to happen, Bitcoin needs a critical mass of adopters. Unfortunately, the Bitcoin feature of capping the number of total BTCs in circulation means that the number of BTC users will always be limited.