Yep. This has been my problem with it since the beginning. A dollar isn't just a dollar. It's a monetary unit that is intrinsically tied into the power and faith of hundreds of millions of people in the idea of a country.
If people stopped believing in America, the dollar would go. How little would have to fail for Bitcoin to have the same problem. I've heard that if the person who owns the large stake in Bitcoin were to transact even one of their coins, all hell would break loose. That doesn't spell stability to me. Neither does "forking" the currency.