If a site doesn't have a certificate , there's no identity verification at all and whether or not to trust the site is left up to the user.
If it does have a certificate, and the certificate doesn't appear to be validated by a CA for the domain or organization, or whatever, then there's a risk of misplaced trust. Anyone can issue a certificate for any domain, but they're only "trustworthy" if they've been signed by a CA.
That's the problem. Why are you deriving trust from a self signed certificate? The UI should be similar to plaintext if there isn't a verifiable chain of trust. There isn't any issue of misplaced trust if you aren't actually labeling the connection as trusted.
> Anyone can issue a certificate for any domain
No, they can issue a certificate that allows for encrypted communication with the current host. Trusting a self signed certificate for any other purpose would be a serious bug.
that's why it's so important to blacklist fast certificate authority that do no domain ownership validation; it's a frail system, but it's the best we got for now.
All of your concerns require an assumption that the browser uses unauthenticated encryption the same as PKI authenticated. Please stop conflating encryption with authentication; they solve different problems. This attitude that a partial solutions should be actively discouraged is why the internet is still uses plaintext which should have been dropped years ago.
encrypted works against MITM because of the certificate trust, if you remove certificate trust from the equation, you'd get the exact opposite result: encrypted would be as secure as plaintext.
What, exactly, are you suggesting is a strawman? I was directly addressing your points.
> encrypted works against MITM because of the certificate trust
Nonsense. Encryption works because of $MATH and a shared secret (or matched pair of secrets) between the two parties communicating (the key or public-key/private-key set). With those elements, communication is protected from 3rd party eavesdroppers. What is not provided is authentication of the 2nd party.
Authentication entirely separate feature. Yes, you should use these two features together whenever possible, as it is very important to both authenticate who you are talking to and protect the conversation from 3rd parties. However, either feature on its own is still better than plaintext.
Yes, without authentication it is possible (and sometimes easy) to MITM an encrypted channel. That does not mean all situations are equal[1]. Self signed certificates can be logged, for example, which can sometimes detect a new or changed MITM. The MITM doesn't have the signing key, which is why the certificate is self signed instead of simply leaving it unsigned.
> encryption would be as secure as plaintext
Security depends on your threat model, and encryption alone protects against traditional non-MITM wiretapping. This includes many forms of mass-surveillance. Just because it is possible to bypass that protection with a MITM doesn't mean you should just give up and send plaintext. (and assuming that everyone can and will get a certificate is delusional; see this very thread for examples) Raising the complexity and cost of an attack is good security.
Yes, the UI should probably report unauthenticated encryption as not trusted, just like plaintext. Also, "secure" is a vague term that is overloaded with multiple meanings which can be misleading. It is better to indicate if something is "authenticated", "protected against eavesdropping", etc.
[1] http://chem.tufts.edu/answersinscience/relativityofwrong.htm
> So you prefer plaintext