TOFU is a pretty bad trust model. Even with SSH you're supposed to validate the fingerprint with some already known trusted source. Otherwise you're just assuming that the first time you connect you aren't being MITMed.
CAs don't make this problem go away; they just shift it to a central "trusted" party. With the CA model, we're assuming that Let's Encrypt isn't being MITMed.
Also we assume no MITM when updating the ca-cert thing on OSes.
In fact the recommended way to do SSH fingerprint checks is
with a CA.