The purpose of Semver is to allow software and people to determine, from the version number alone, whether and what has changed. But even though many projects claim adherence to the Semver standard nowadays, the ecosystem as a whole isn't, and never has been, anywhere close to the point where a reasonable engineer would actually rely on version numbers to decide whether to upgrade.
Imagine you see that one of your library dependencies, which professes adherence to Semver, has released version 1.2.3, up from version 1.2.2 which your project currently uses. Would you feel comfortable doing the update, without checking whether the project still compiles and passes its test suite afterwards?
If you answered "no", then you are basically admitting that Semver is worthless in practice. If you are going to test everything anyway, version numbers might as well be a simple incrementing integer, or a date, or a random string. Upgrading will still consist of trying the new version, and, on failure, rolling back – which is exactly what you are probably already doing anyway.
Semver doesn't work because you can't actually trust that all your dependencies have verified their claimed Semver compliance to the extent that would be necessary for your project to rely on it. You have to do that verification yourself, which makes the Semver version number at best an occasionally useful hint. And that's not even talking about all the other problems, such as different projects having different interpretations of what constitutes a "breaking" change, and almost none of the older, established libraries even claiming to follow Semver in the first place.