That makes sense. I wonder if any web servers have smarter decision making for that. Like give a hard 301 to modern browsers, but let older or nonstandard clients get a standard http response.
Besides, if the security negotiation is to be done in plaintext, then it's trivial for an attacker to MITM a connection, replace the User-Agent headers, and then trick a server into thinking it should serve content insecurely. This is a huge gaping attack vector. It's better to just always serve securely.