I think criminals gaining access to my bank account is far more lucrative for them than gaining access to my social security account so I don't think it makes the problem particularly worse than it already is.

But besides that, the federal government already has an SSO provider that supports smart cards, U2F, and TOTP. I suspect many agencies will choose to adopt that rather than trying to roll their own SMS auth.