When I worked at a defense contractor, we were told to remove our badges when leaving the office, lest they be seen (and somehow copied) by spies hanging out at nearby restaurants. I would have thought any federal agency concerned about security would have the same policy.

Perhaps the policy was a little paranoid, but I also can't think of any reason to wear a badge beyond the area where it's required.