If a project actually follows semver, then you know that unless the first number changes, there won't be any (significant) backwards-incompatible changes - eg, your config files will still work as expected.
This is the major obstacle with semver. People aren't terribly psyched to increase the major version whenever incompatibilities happen, because other people have different expectations of major version numbers.