Physics. Entropy.
If it absorbed energy that touched it, it would heat up. You could then use the difference between it and its cooler surroundings to run an engine. Waste heat from the engine could then be absorbed to keep the engine running. You'd have created a perpetual motion machine, which is considered impossible.
In a closed system of a star, the sphere, and the outer space eventually energy will flow from the star to the outer space region.
There's no perfect insulation. (See also Maxwell's demon.)
Of course detection is a different matter altogether. Hiding better and better is possible. But if you really want to hide a Dyson swarm is not the best way :)