That strikes me as exactly the sort of guarantee that's _not_ made by semver. You can rely on that behavior, but then you have to check against every patch version, because that's exactly the sort of thing that could change out from under you. Maintainers shouldn't worry about breaking this case- they never promised to support it in the first place.
I think a lot of those cases should be a major version change, and that's totally fine. We _should_ minimize the number of breaking changes to code people depend on, but sometimes the best fix to the (totally reasonable) situations you outlined above is to make a new major version. Users will have to make changes to upgrade (instead of just bumping the versions), but it won't change out from under them. Some docs about what's changing, why, and how to migrate your code go a long way here.