I agree with you that ASLR, NX, and CFI are the most important system level defenses to employ.
This assertion confuses me.
I use fail2ban on boxes I have key-only ssh configured for.
Are you aware fail2ban works for services other than ssh?
If an attacker / script knocks unsuccessfully on my ssh door, other doors are then closed to them.
I also get much (much!) cleaner logs thanks to fail2ban.
I suspect that you're confusing fail2ban and port-knocking (or using fail2ban as a port-knocker).
The point of fail2ban is to prevent an attacker from brute-forcing your server. In a key-only config, the chances of getting brute forced is smaller (by a few orders of magnitude) than getting hit by an asteroid and having the server get hit by an asteroid, so fail2ban doesn't really help.
_In theory_, the same would be true for port-knocking.
However, in practice, sshd can have security holes which a malicious scanner could exploit. And while port-knocking doesn't help against a determined attacker (it's subject to MITM, replay-attacks), it does help with defense-in-depth.
Too bad that sshd can't enforce use of password-proctected keys on the server side..
My point is that in general it would be better to disable password auth and only use key based auth, but only if you could somehow guarantee that the users wouldn't do crazy things like use password-less keys. But as you can't do that on the server-side, what other options do you have?
It's about reaction of the staff to key leak:
>> A HPC center [...] disabled key logins IIRC due to some incident where an attacker had got hold of a password-less key.
This reaction seems just silly.