Its a big privacy leak, not a big security leak.
Your Pubkey can be used to cross-match multiple identities. Example: You have different coding personae. One that is activist, one that is company-peon. Different accounts, same SSH pubkey in Github or other server with publicly listed pubkeys --> Same person confirmed.
As a result of this the information can be used to target each of the identities in a more precise manner. On the human layer of the security side: New phishing/deception/blackmail vectors.
On the organizational layer: we have to target these keybearer devices now.
Maybe it even helps in a cryptanalytic way in some weird exotic scenario but not substantially.
And of course separation of concerns if you have different keybearer devices.
(Also the famous Keysticks are a nice solution to that organizationally but they are an additional risk for big scale attacks by having biased RNGs. In the end its hardware and audits are just a voluntary thing by corps. They can always choose to hide things from auditors or do a compromised batch at their mercy.)