Your computer can't simply remember the certificate that was used for the last connection, because then sites can never change their certificates (which would be a huge issue).
Moxie mentioned before that using notaries would work, of course you have to worry about notaries being compromised but that's a much easier threat model.