But access to the technology to transfer bitcoin does not require infrastructure and trust. I can send you a bitcoin without needing a bank or any other large authority. Moreover, I can do so relatively anonymously, and from anywhere in the world. It is therefore much, much harder to impose restrictions on moving bitcoin than moving USD or other non-digital currencies, because there are no large authorities capable of enforcing those restrictions.
(That is, the government can of course pass laws saying I can't send you bitcoin, or limiting the circumstances in which I could. But the most they can do is punish me if they can catch me - they can't actually stop it. Whereas banks do have the power to actually stop me from sending you USD. [Unless I want to do so physically, which is hard to do in quantity or over any distance.])