Why can't I use TLS/SSL to deliver the Javascript crypto code?
You can. It's harder than it sounds, but you safely transmit Javascript crypto to a browser using SSL. The problem is, having established a secure channel with SSL, you no longer need Javascript cryptography; you have "real" cryptography.
So, you can deliver the JS to the browser securely!