For better or worse, gender is frequently used as a piece of identifying information.
This could cause problems.
For example, many cisgendered privacy oriented folks could choice this option.
The possible result would a very large chunk of the population having no gender marker on their passport.
This might sound good - but it would make it very hard to study, for example, whether men or women receive preferential treatment from US Consuls. Or whether men or women face different threats abroad (some embassies keep statistics on sexual assaults).
If we hypothetically assume that gender is no longer recorded, that means gender inequality becomes very difficult to study.
Additionally, US Passports function as identity documents for US Citizend abroad. When I travel, I use it to pick up prescriptions. Most foreign pharmacies will accept it, but I occasionally run into trouble.
I simply hope the State Department looked at these issue.