> Allowing transparent downgrades of self-signed certificates would be a big security hole.
Automatically generated self-signed certificates should have replaced all plaintext HTTP 15-20 years ago. The big security hole was allowing passive surveillance, ISP-level page injection vandalism[1]/attacks[2].
The web could have been almost completely protected from several classes of attack a decade ago, but this stupid insistence on conflating protection from 3rd part eavesdropping or corruption during transit with the authentication of the server. These are entirely separate problems that do not need to be solved at the same time.
> I am requiring the script to be served securely
You're requiring it to be served over HTTPS, which doesn't necessarily mean "secure", because "secure" covers several different goals. You're also strongly trusting the PKI system. Do you trust all the certificate authorities your browser includes by default?
Of course, because HTTP still exists, the initial request for the HTML that contains your <script> tag could be sent plaintext and thus modified during transit in many different ways.
> serve a malicious script to my users.
That can still happen without proper pinning, or if the local browser downgrades the request back to HTTP. Unfortunately this isn't particularly uncommon with corporate/school proxy, in-flight wi-fi services that forge certificates[3], and Superfish-style junk all removing both the encryption and the authentication provided by TLS.
Regarding your specific example about loading Javascript referenced in an HTML document's <script> tag, the solution is to validate the data, not the server. The valid server can still send incorrect data. If you include hashes about a page's subresources[4], the browser can validate the integrity of the file it received.
[1] https://arstechnica.com/tech-policy/2014/09/why-comcasts-jav...
[2] https://citizenlab.ca/2015/04/chinas-great-cannon/
[3] https://arstechnica.com/information-technology/2016/02/why-y...
[4] https://www.w3.org/TR/SRI/