Do you ever examine logs? Fail2ban (or even better, using a nonstandard port or both) keep the noise out of lots to the point that you can still examine them manually.
Without it, 99.99% of the average ssh log is failed back attempts.
Without it, 99.99% of the average ssh log is failed back attempts.