I think the game may have to change a few times already before a Dyson swarm (or even a significant fraction of one) could be built. We're going to need cheap clean power in large quantities to even think about manufacturing something like that.

You have to walk before you can run.