You just ensure the browser warns you if the self-signed certificate changes.
It works fine this way for ssh.
It works fine this way for ssh.
There are loads of ways this could go wrong (the phishing website earnestly telling the user that they "changed" their SSL key and that you need to follow these simple steps to "fix" your browser; the initial contact being the bogus website; the huge loss of traffic if your SSL key legitimately changes and you're not prepared; you need to deal with revocation).
It works for ssh because you generally know the parties you are trying to connect to ahead of time and have ways to communicate with them (to confirm keys etc).
You do not have the same relationship with 99.99% of the servers you will be connecting to over https. It does not work.