The very problem here is that I am not ok with middle man becoming the real website. If you are ok with that, you don't have a problem. You can use TOFU. Good for you. But I have a problem with that. So I can't use TOFU.
What about TLS certificates attested by CAs who validate the real world legal entity? Would you agree that this is a solved problem there?