The issue in Rust isn't that semver isn't doing the right thing for us. It's that every so often (maybe as rarely as 2-3 times per year for the entire Rust community!) there's a spectacularly painful and regrettable accidental semver break that causes tons of effort to be spent by both maintainers and many downstream users.
For every such incident, there are probably thousands of little semver violations here and there that we don't notice and where nobody is affected by them.
cargo-semver-checks is about letting maintainers have actionable information at the right time so they can engage in well-informed decision-making. Without this tool, the alternative is: every time you publish a new version, roll a d100; on a roll of 2 or 3, you broke semver but nobody noticed; on a roll of 1, you wake up to a GitHub issue with 1000 other issues linked to it that says "you broke semver, please fix ASAP." This is the part that sucks! (The numbers are approximately accurate based on our study of the top 1000 most-downloaded crates.) The remaining 97 times, everything is great -- you publish a new version, everyone happily uses it, astounding success all around.
You can think of cargo-semver-checks as an automated way to answer a series of queries like "are there any public functions that existed in the previous version and don't exist in the new version?" Any results produced by any of those queries are examples of breaking changes.
This is why our "more than 1 in 6" study found so many previously-unreported semver violations — in our study we were able to find issues in all sorts of dark corners of library APIs, then generate programs specifically designed to rely on the affected behavior in a way that works fine in one API version and causes a compilation error in the subsequent (non-major) version.
that's exactly my point, without this great tool he can't know whether semver is broken from his practical use since his use is not exhaustive
I'm glad to hear of your positive experience with semver. Unfortunately, the issue with semver violations is that their impact has extremely high variance, easily ranging between "nobody noticed" and "ecosystem-wide pain":
- Most of the semver violations we found appear to have never been reported or noticed. After talking to a representative subset of the maintainers of those crates, they were generally surprised to find out a semver violation had happened.
- At the same time, a quick search for "violated semver" in Rust repos on GitHub will surface hundreds of issues, many of them heavily linked-to by issues in other projects. Every such issue represents dozens or hundreds of cumulative engineer-hours of triage and remediation in both the project where the semver accident happened and in affected downstream projects. All this work is regrettable, in the sense that both maintainers and downstream consumers would have preferred if it could have been avoided.
Most semver violations aren't intentional -- they are accidents that happen to all of us regardless of skill level, carefulness, experience, etc. Most of the time, those accidents go unnoticed or have minimal impact. Every so often, they trigger nearly ecosystem-wide consequences for a few days or weeks. (Some examples on my blog: https://predr.ag/blog/toward-fearless-cargo-update/ and https://predr.ag/tags/semver ) And sometimes, breaking semver on purpose can be the correct thing to do!
cargo-semver-checks aims to be a tool such that maintainers never have to say "if I had known this API change had happened, I wouldn't have published this version." Its biggest impact should be at minimizing or altogether preventing those ecosystem-wide semver issues that happen from time to time. Our day-to-day experience with semver should remain unchanged otherwise.