I don't think there's any circle you can draw around federal buildings that wouldn't include residential buildings also.

If you declared all federal buildings to be part of the district (and everything else Maryland) it would get messy fast:

+ What about commercial buildings that the feds lease? (There are many of these.)

+ What about federal buildings which are leased to commercial organizations? (Not sure if there are any, but it wouldn't surprise me.)

+ Would federal DC have its own first responders specifically to service federal buildings?

+ Would DC pay Maryland taxes for road maintenance between its "islands" of office space?

Etc.