What's the history behind this? I doubt there was legacy or backwards compat reasoning? Allowing for such a loose and wide interpretation makes for complicated parsing and numerous exploits.
> for complicated parsing
Somewhat
> numerous exploits.
Nah. It's mostly localized to a string to hostname processing, ie it never occurs in the network stack and happens on the user'side of things and permissions (think CLI and interpreted languges)
As an example, think of a cloud based web performance monitoring system. I trick it into making HTTP requests to 169.254.169.254, and I get access to data from their AWS metadata service...