The current embassy is indeed in DC proper, but it isn't directly across the river from DCA.

It's about a mile upriver, near the Watergate.