What's the practical value you're hoping to gain by hard-enforcing semver, especially if the surface area of HTTP APIs is a lot smaller? And this isn't a loaded question. I might just not be privy to your team's needs.
Also consider (and this again falls within the realm of personal opinion) the value of achieving _maximum possible statelessness_ as a developer. If I'm reading you right, you're asking me to know and remember, and probably write down somewhere, that your API's versioning means something different than others. So I now have another branching path to maintain: how to behave when your API changes. What I like about the simple numbered API schema is that I can come back to the API usage in a stateless manner. I just see that the number is different and I know what to do: learn what changed and decide if I have to act on it. Of course I could do this by looking at a semver as a unique identifier without semantic meaning, but that kind of leads to my initial point: encourage best practice by not giving the developer a semantic shortcut to which they can abdicate responsibility.