Would HSTS have helped in this situation?

HSTS is mainly to prevent SSL-stripping. But I think part of HSTS could also note that the certificate was trusted, and then having an HSTS header could entirely prevent any later connection with the self-signed certificate, without clearing the HSTS history.

You may not need to even store the extra bit, it's enough to say if you have HSTS then by default the connection must not just be encrypted, but it must be trusted.

Do current browsers entirely prevent a connection to untrusted certs when HSTS is set? Or is it just the same error you get when connecting to any self-signed cert?