As an American, it is not true that most people don't have passports.

There are currently 180 million and change active issued passport to US citizens.