The US Embassy compound is on the Akasaka side of Roppongi and the Embassy itself is towards Toranomon. There is also a small US Army base between Roppongi and the Aoyama graveyard, right next to the new National Museum.

FWIW, it's not just the Japanese that can not get into that base, civilian US citizens will also probably find trouble getting in. Which is a bummer since the PX there sells affordable cuts of meat.