Or they have your discarded dead drive from your backup disk that you forgot to enable full disk encryption on. Or they rooted a machine that you were using "ssh -A" on to forward your agent, and have been able to temporarily hijack your SSH agent session. Or there's an information disclosure vulnerability in your web browser that allows them to read files from your hard drive, though without control of what files they can read, and by chance the attacker got lucky and one of the files was .ssh/id_rsa.
While the above description goes way further than I probably would, and yeah, 2FA would be more reasonable, there are good reasons to practice defense in depth, and try to design systems that have a chance of staying secure even if one of your assumptions fails like "private key is compromised".
Someone brute forcing said passphrase? Yeah no.
2 factor with a token is ideal, I suppose I should have added that to the list, but that is outside of feasibility for most users.
Still, for someone who doesn't have RSA tokens etc available, assuming your remote machine doesn't get owned and someone doesn't extract your pass-phrase with a wrench, that list makes for a very secure system.
You should be auditing everything already, and adding in sudo now adds another attack vector. You also now have two different accounts that can be manipulated to compromise a system.