You visited the udvar-hazy center which is out by Dulles airport in Chantilly. It is still part of the smithsonian, but more focused on displaying lots of different aircraft in a few large hangers. Very impressive and worth a visit, but not the main air and space facility.
Either you had a legitimate miscommunication, or you got scammed by your driver. No one from around the DC metro would confuse the two.