Geography, for a start. Federal buildings quite widespread, not just concentrated around the mall.
If that doesn't work, it doesn't really matter that much if you sweep up a few commercial buildings into DC. Most of those federal buildings are mixed in with retail, hotels, convention centers, offices, and the like, which doesn't cause disenfranchisement if you include them within DC. The big problem is residential areas.
If you declared all federal buildings to be part of the district (and everything else Maryland) it would get messy fast:
+ What about commercial buildings that the feds lease? (There are many of these.)
+ What about federal buildings which are leased to commercial organizations? (Not sure if there are any, but it wouldn't surprise me.)
+ Would federal DC have its own first responders specifically to service federal buildings?
+ Would DC pay Maryland taxes for road maintenance between its "islands" of office space?
Etc.
Once you walked off the sidewalk, or off the third floor elevators, or through that door, etc, you would be on "federal turf" so to speak. It's not about a geographical circle.
The core national mall area is already protected by US Park Police, it would be possible to create a core DC area as a national park and return the rest of all land to the states. Federal buildings on land returned to the states would be treated like any other federal building in a state. There are plenty of federal buildings in NoVa and Maryland already.