4,328 karma · joined August 11, 2015
- GitHub: https://github.com/obi1kenobi/trustfall
- conference talk: https://www.hytradboi.com/2022/how-to-query-almost-everything
- web demo: https://play.predr.ag/hackernews
Creator of `cargo-semver-checks`, a linter for semantic versioning of Rust crates: https://crates.io/crates/cargo-semver-checks
Blog: https://predr.ag/blog/
Mastodon: `@predrag@hachyderm.io`
Twitter: https://twitter.com/PredragGruevski
Talks: https://predr.ag/talks/
I'm sure someone could build a semver-checker for Python or JS or Java or whatever, and I'd even be willing to help them and walk them through how cargo-semver-checks works. But in Rust, a big driver of adoption is that Rust and cargo are opinionated about things, so people adopt cargo-semver-checks not necessarily out of love for semver but out of a desire to not wake up to 100 people being upset in their GitHub issues about broken builds. The adoption situation might be harder for other languages with less opinionated versioning stories.
No, as it turns out it very much is not even close to the same amount of legwork.
> I read this as
I don't think your attempt at a re-statement is accurate to the source. Also, "regardless of versioning strategy" is simply not an option in Rust, since the entire ecosystem is opinionated about how versions should work and overall everything works very well.
This post is about taking something that is already very rare (though painful when it does happen!) and making it even more rare. It shows a concrete way to do so, and also shows that so far we've been "getting lucky" by experiencing breakage less often than we might have been, given the same code changes.
In other words: `cargo update` today has a tiny chance of breaking your Rust project. But if everyone used `cargo-semver-checks` every time, as demonstrated by what the blog post's analysis caught, we'd reduce the number of accidentally-breaking releases by *3% of all releases*. That is an astoundingly high reduction, so coupled with good testing practices and other static checks (type system, borrow checker, etc.) it means that `cargo update` will be even more astonishingly unlikely to break your project.
If semver suggests a release isn't breaking, in Rust it overwhelmingly isn't breaking. Around 3% of the time, assuming you use every obscure bit of public API in the library, you might get broken by a release when you shouldn't be. In practice, that percentage is way lower than that: nobody uses the entire public API of anything, or comes anywhere close. That's why the post says these semver violations were unknown, even though most of them were multiple years old. If something breaks, and you weren't using it anyway, you were not broken.
The post is emphatically in favor of semver as used by Rust, because it has clear benefits and works great the vast majority of the time. The part that we want to make better is to lower the odds even further that an accidental breaking change breaks a bunch of people's code and ruins their day.
No single technique can do that: not testing, not fuzzing, not cargo-semver-checks. But a sound combination of techniques can drive the odds to damn near zero for all practical purposes.
So I'm a bit confused about the position you seem to be arguing against, when the tool and the entire ecosystem are doing exactly what you (AFAICT) consider a good idea.
Rust generally has more exceptions than most languages about which breaking changes are technically considered non-major, so writing such guidelines for Python or Java shouldn't be too difficult.
The same tech and ideas that power `cargo-semver-checks` could also be repurposed for those languages as well. If a company is interested in sponsoring such work, I'd be happy to help build something like that!
Sometimes it isn't fine, of course. Those cases fortunately are few enough and not damaging enough that this approach hasn't been that problematic thus far.
> Knowing that something is _definitely_ a breaking change because an API was removed or modified in an incompatible way will still save you time.
We want to help maintainers spot accidental breaking changes early — before they are released — instead of both maintainers and downstream projects having to scramble for fixes after an accidental breaking change is released.
- cargo-semver-checks: https://github.com/obi1kenobi/cargo-semver-checks
- Trustfall query engine, which powers cargo-semver-checks: https://github.com/obi1kenobi/trustfall
- Trustfall playground, where you can query Rust library APIs in your browser -- for example, "which structs in `itertools` are importable by more than one path": https://play.predr.ag/rustdoc#?f=2&q=*3-Structs-importable-f...
- 10min conference talk on Trustfall: https://www.hytradboi.com/2022/how-to-query-almost-everythin...
I'm also giving a talk at P99 CONF in a few months about how Trustfall's new optimizations API made cargo-semver-checks over 2300x faster: https://twitter.com/PredragGruevski/status/16893002495908003...
What we did:
1. Scan Rust's most popular 1000 crates with `cargo-semver-checks`
2. Triage & verify 3000+ semver violations
3. Build better tooling instead of blaming human error
Around 1 in 31 releases had at least one semver violation.
More than 1 in 6 crates violated semver in at least one release.
These numbers aren't just "sum up everything `cargo-semver-checks` reported." We did a ton of validation through a combination of automated and manual means, and a big chunk of the blog post is dedicated to talking about that.
Here's just one of those validation steps. For each breaking change, we constructed a "witness," a program that gets broken by it. We then verified that it:
- fails to compile on the release with the semver-violating change
- compiles fine on the previous version
Along the way, we discovered multiple `rustc` and `cargo-semver-checks` bugs, and found out a lot of interesting edge cases about semver. Also, now you know another reason why it was so important to us to add those huge performance optimizations from a few months ago: https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov...
`cargo-semver-checks` by default doesn't check for any semver obligations for code behind such features. (I'm its maintainer )
The issue in Rust isn't that semver isn't doing the right thing for us. It's that every so often (maybe as rarely as 2-3 times per year for the entire Rust community!) there's a spectacularly painful and regrettable accidental semver break that causes tons of effort to be spent by both maintainers and many downstream users.
For every such incident, there are probably thousands of little semver violations here and there that we don't notice and where nobody is affected by them.
cargo-semver-checks is about letting maintainers have actionable information at the right time so they can engage in well-informed decision-making. Without this tool, the alternative is: every time you publish a new version, roll a d100; on a roll of 2 or 3, you broke semver but nobody noticed; on a roll of 1, you wake up to a GitHub issue with 1000 other issues linked to it that says "you broke semver, please fix ASAP." This is the part that sucks! (The numbers are approximately accurate based on our study of the top 1000 most-downloaded crates.) The remaining 97 times, everything is great -- you publish a new version, everyone happily uses it, astounding success all around.
You can think of cargo-semver-checks as an automated way to answer a series of queries like "are there any public functions that existed in the previous version and don't exist in the new version?" Any results produced by any of those queries are examples of breaking changes.
This is why our "more than 1 in 6" study found so many previously-unreported semver violations — in our study we were able to find issues in all sorts of dark corners of library APIs, then generate programs specifically designed to rely on the affected behavior in a way that works fine in one API version and causes a compilation error in the subsequent (non-major) version.
I'm glad to hear of your positive experience with semver. Unfortunately, the issue with semver violations is that their impact has extremely high variance, easily ranging between "nobody noticed" and "ecosystem-wide pain":
- Most of the semver violations we found appear to have never been reported or noticed. After talking to a representative subset of the maintainers of those crates, they were generally surprised to find out a semver violation had happened.
- At the same time, a quick search for "violated semver" in Rust repos on GitHub will surface hundreds of issues, many of them heavily linked-to by issues in other projects. Every such issue represents dozens or hundreds of cumulative engineer-hours of triage and remediation in both the project where the semver accident happened and in affected downstream projects. All this work is regrettable, in the sense that both maintainers and downstream consumers would have preferred if it could have been avoided.
Most semver violations aren't intentional -- they are accidents that happen to all of us regardless of skill level, carefulness, experience, etc. Most of the time, those accidents go unnoticed or have minimal impact. Every so often, they trigger nearly ecosystem-wide consequences for a few days or weeks. (Some examples on my blog: https://predr.ag/blog/toward-fearless-cargo-update/ and https://predr.ag/tags/semver ) And sometimes, breaking semver on purpose can be the correct thing to do!
cargo-semver-checks aims to be a tool such that maintainers never have to say "if I had known this API change had happened, I wouldn't have published this version." Its biggest impact should be at minimizing or altogether preventing those ecosystem-wide semver issues that happen from time to time. Our day-to-day experience with semver should remain unchanged otherwise.
If you ever feel like doing more work in the space, I'd love to work together on cargo-semver-checks!
If you or anyone else is interested in building such a tool, ping me and I'd be happy to help you get started.
The goal of this tool is to aid in preventing regrettable accidents like that, by informing maintainers about semver issues before they are published.
Running `cargo publish` would check for semver issues and alert if it finds anything. Users would be able to override it, of course, but most often they'd probably accept the version change or revert the semver break if it wasn't intended.
For Rust folks, I'm talking about auto traits like Send and Sync.
More info and examples here:
https://predr.ag/blog/toward-fearless-cargo-update/
TL;DR: Semver is particularly tricky in Rust and can be broken in many far-from-obvious ways. When a library breaks semver it can cause compile errors in every downstream bit of code that uses it. This sucks for everyone on all sides.
https://news.ycombinator.com/item?id=36653954
One would just need a machine readable data source for Go package APIs, and would plug it into the Trustfall query engine. I'm not a Go expert but most languages I've worked with have something like that.
If you might be interested in building this for Go, ping me and I'd be happy to help you hit the ground running!
Trustfall is even more general than semver actually, one can use it to query any data source -- even HN itself:
https://play.predr.ag/hackernews
Here's a link to that project if you'd like to check it out:
https://github.com/obi1kenobi/trustfall
I gave a 10min conference talk on it last year titled "How to query (almost) everything"
https://www.hytradboi.com/2022/how-to-query-almost-everythin...
This is obviously a lower bound, and yet it's quite a high number already.
I plan to write more about this on my blog in the coming weeks:
In the meantime I put some more info (including a link to slides from a recent talk I gave on this) in another comment:
I've written several blog posts about how semver in Rust is particularly tricky and has unexpected edge cases, if you'd like to see specific examples of things non-obviously going wrong:
Also, I've been working with the cargo team and the plan is to merge cargo-semver-checks into cargo itself, so it runs automatically on `cargo publish`. This would be similar to how cargo checks for uncommitted git changes: it will alert if it finds anything but you can override it if you think that's warranted.
I recently gave a talk about the tool, addressing:
- why semver in Rust is important and harder than it looks
- how the tool works
- why it took until 2022-2023 to solve this problem
There's no recording but here are the slides: https://docs.google.com/presentation/d/1gVYM9-YLrBmRYSuHqW8r...
A lot of semver is just queries over APIs. For example: "find public functions that existed in the last version but don't exist in the new one." Any such function we find is a breaking change.
Cargo-semver-checks uses an engine designed to query any data sources called Trustfall, and uses rustdoc JSON (which has an unstable format that changes ~once a month).
Here's a query playground where you can try running queries over rustdoc:
More info about all this on my blog:
A few of my more popular posts:
- Falsehoods programmers believe about undefined behavior: https://predr.ag/blog/falsehoods-programmers-believe-about-u...
- To ace exams, get better at the easy questions: https://predr.ag/blog/to-ace-exams-get-better-at-the-easy-qu...
- Speeding up Rust semver-checking by over 2000x: https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov...
- Mediocrity can be a sign of excellence: https://predr.ag/blog/mediocrity-can-be-a-sign-of-excellence...
- Breaking semver in Rust by adding a private type, or by adding an import: https://predr.ag/blog/breaking-semver-in-rust-by-adding-priv...
- Debugging Safari: If at first you succeed, don't try again: https://predr.ag/blog/debugging-safari-if-at-first-you-succe...