Not illegal (even though it should be) unless they prevent you from flying without it. Currently you can go through security with no identification at all as long as you answer their questions and they can verify your citizenship some way (which can add a few hours to your tenure at the airport).