HNHacker News
TopNewBestAskShowJobs

obi1kenobi

4,328 karma · joined August 11, 2015

Building Trustfall, a query engine able to query any dataset:

- GitHub: https://github.com/obi1kenobi/trustfall

- conference talk: https://www.hytradboi.com/2022/how-to-query-almost-everything

- web demo: https://play.predr.ag/hackernews

Creator of `cargo-semver-checks`, a linter for semantic versioning of Rust crates: https://crates.io/crates/cargo-semver-checks

Blog: https://predr.ag/blog/

Mastodon: `@predrag@hachyderm.io`

Twitter: https://twitter.com/PredragGruevski

Talks: https://predr.ag/talks/

submissionscomments
obi1kenobi··on Semver violations are common, better tooling is the answer
Thank you! I love mentoring people, and it was a privilege to work with these particularly talented and hard-working undergrads.
obi1kenobi··on Semver violations are common, better tooling is the answer
I'm curious too! And I also agree that semver in _most_ languages is a mess.

I'm sure someone could build a semver-checker for Python or JS or Java or whatever, and I'd even be willing to help them and walk them through how cargo-semver-checks works. But in Rust, a big driver of adoption is that Rust and cargo are opinionated about things, so people adopt cargo-semver-checks not necessarily out of love for semver but out of a desire to not wake up to 100 people being upset in their GitHub issues about broken builds. The adoption situation might be harder for other languages with less opinionated versioning stories.

obi1kenobi··on Semver violations are common, better tooling is the answer
> Isn't it the same release-verification legwork either way?

No, as it turns out it very much is not even close to the same amount of legwork.

> I read this as

I don't think your attempt at a re-statement is accurate to the source. Also, "regardless of versioning strategy" is simply not an option in Rust, since the entire ecosystem is opinionated about how versions should work and overall everything works very well.

This post is about taking something that is already very rare (though painful when it does happen!) and making it even more rare. It shows a concrete way to do so, and also shows that so far we've been "getting lucky" by experiencing breakage less often than we might have been, given the same code changes.

In other words: `cargo update` today has a tiny chance of breaking your Rust project. But if everyone used `cargo-semver-checks` every time, as demonstrated by what the blog post's analysis caught, we'd reduce the number of accidentally-breaking releases by *3% of all releases*. That is an astoundingly high reduction, so coupled with good testing practices and other static checks (type system, borrow checker, etc.) it means that `cargo update` will be even more astonishingly unlikely to break your project.

obi1kenobi··on Semver violations are common, better tooling is the answer
As a co-author of the post, I wouldn't agree with that summary.

If semver suggests a release isn't breaking, in Rust it overwhelmingly isn't breaking. Around 3% of the time, assuming you use every obscure bit of public API in the library, you might get broken by a release when you shouldn't be. In practice, that percentage is way lower than that: nobody uses the entire public API of anything, or comes anywhere close. That's why the post says these semver violations were unknown, even though most of them were multiple years old. If something breaks, and you weren't using it anyway, you were not broken.

The post is emphatically in favor of semver as used by Rust, because it has clear benefits and works great the vast majority of the time. The part that we want to make better is to lower the odds even further that an accidental breaking change breaks a bunch of people's code and ruins their day.

No single technique can do that: not testing, not fuzzing, not cargo-semver-checks. But a sound combination of techniques can drive the odds to damn near zero for all practical purposes.

obi1kenobi··on Semver violations are common, better tooling is the answer
cargo-semver-checks checks almost entirely only major.minor versioning issues. Rust's flavor of semver doesn't impose or demand any particular rules about what is a minor and what is a patch version.

So I'm a bit confused about the position you seem to be arguing against, when the tool and the entire ecosystem are doing exactly what you (AFAICT) consider a good idea.

obi1kenobi··on Semver violations are common, better tooling is the answer
All those rules are derived from a much smaller set of rules that are easier to understand. In Rust, those rules are more or less "breaking changes require a major version, unless the breaking change could have been avoided by the caller by adding disambiguation ahead of time." More details here: https://predr.ag/blog/some-rust-breaking-changes-do-not-requ...

Rust generally has more exceptions than most languages about which breaking changes are technically considered non-major, so writing such guidelines for Python or Java shouldn't be too difficult.

The same tech and ideas that power `cargo-semver-checks` could also be repurposed for those languages as well. If a company is interested in sponsoring such work, I'd be happy to help build something like that!

obi1kenobi··on Semver violations are common, better tooling is the answer
Speaking for myself in Rust, I blindly update everything and fix what's broken. My projects tend to have substantial test suites, so if the compiler thinks it's fine and the tests think it's fine, it's probably fine.

Sometimes it isn't fine, of course. Those cases fortunately are few enough and not damaging enough that this approach hasn't been that problematic thus far.

obi1kenobi··on Semver violations are common, better tooling is the answer
Of course! `cargo-semver-checks` is not a replacement for testing. It's exactly the kind of tool you describe:

> Knowing that something is _definitely_ a breaking change because an API was removed or modified in an incompatible way will still save you time.

We want to help maintainers spot accidental breaking changes early — before they are released — instead of both maintainers and downstream projects having to scramble for fixes after an accidental breaking change is released.

obi1kenobi··on Semver violations are common, better tooling is the answer
If you'd like to dig deeper, here are some links:

- cargo-semver-checks: https://github.com/obi1kenobi/cargo-semver-checks

- Trustfall query engine, which powers cargo-semver-checks: https://github.com/obi1kenobi/trustfall

- Trustfall playground, where you can query Rust library APIs in your browser -- for example, "which structs in `itertools` are importable by more than one path": https://play.predr.ag/rustdoc#?f=2&q=*3-Structs-importable-f...

- 10min conference talk on Trustfall: https://www.hytradboi.com/2022/how-to-query-almost-everythin...

I'm also giving a talk at P99 CONF in a few months about how Trustfall's new optimizations API made cargo-semver-checks over 2300x faster: https://twitter.com/PredragGruevski/status/16893002495908003...

obi1kenobi··on Semver violations are common, better tooling is the answer
Post co-author + `cargo-semver-checks` maintainer here, AMA.

What we did:

1. Scan Rust's most popular 1000 crates with `cargo-semver-checks`

2. Triage & verify 3000+ semver violations

3. Build better tooling instead of blaming human error

Around 1 in 31 releases had at least one semver violation.

More than 1 in 6 crates violated semver in at least one release.

These numbers aren't just "sum up everything `cargo-semver-checks` reported." We did a ton of validation through a combination of automated and manual means, and a big chunk of the blog post is dedicated to talking about that.

Here's just one of those validation steps. For each breaking change, we constructed a "witness," a program that gets broken by it. We then verified that it:

- fails to compile on the release with the semver-violating change

- compiles fine on the previous version

Along the way, we discovered multiple `rustc` and `cargo-semver-checks` bugs, and found out a lot of interesting edge cases about semver. Also, now you know another reason why it was so important to us to add those huge performance optimizations from a few months ago: https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov...

obi1kenobi··on Semver violations are common, better tooling is the answer
In Rust, this is often done by using optional features, and usually features named like `unstable-<something>`. They are a form of conditional compilation: people that don't opt in don't see that code, and people that do opt in know the code is unstable.

`cargo-semver-checks` by default doesn't check for any semver obligations for code behind such features. (I'm its maintainer )

obi1kenobi··on I claim Rich Hickey is wrong about non-null arguments to functions (2020)
Interestingly, even though it's a breaking change, breaking inference is explicitly defined to not be a semver-major change in Rust: https://predr.ag/blog/some-rust-breaking-changes-do-not-requ...
obi1kenobi··on Cargo-semver-checks: Scan your Rust crate for semver violations
Both can be true at the same time, and I believe that to be the case.

The issue in Rust isn't that semver isn't doing the right thing for us. It's that every so often (maybe as rarely as 2-3 times per year for the entire Rust community!) there's a spectacularly painful and regrettable accidental semver break that causes tons of effort to be spent by both maintainers and many downstream users.

For every such incident, there are probably thousands of little semver violations here and there that we don't notice and where nobody is affected by them.

cargo-semver-checks is about letting maintainers have actionable information at the right time so they can engage in well-informed decision-making. Without this tool, the alternative is: every time you publish a new version, roll a d100; on a roll of 2 or 3, you broke semver but nobody noticed; on a roll of 1, you wake up to a GitHub issue with 1000 other issues linked to it that says "you broke semver, please fix ASAP." This is the part that sucks! (The numbers are approximately accurate based on our study of the top 1000 most-downloaded crates.) The remaining 97 times, everything is great -- you publish a new version, everyone happily uses it, astounding success all around.

obi1kenobi··on Cargo-semver-checks: Scan your Rust crate for semver violations
cargo-semver-checks is not dependent on whether code paths are used, because it doesn't analyze from the "use" side. It analyzes from the "what API is exposed" side.

You can think of cargo-semver-checks as an automated way to answer a series of queries like "are there any public functions that existed in the previous version and don't exist in the new version?" Any results produced by any of those queries are examples of breaking changes.

This is why our "more than 1 in 6" study found so many previously-unreported semver violations — in our study we were able to find issues in all sorts of dark corners of library APIs, then generate programs specifically designed to rely on the affected behavior in a way that works fine in one API version and causes a compilation error in the subsequent (non-major) version.

obi1kenobi··on Cargo-semver-checks: Scan your Rust crate for semver violations
Tool and study coauthor here. The "more than 1 in 6" ratio is purely in the top 1000 most-downloaded crates on crates.io, which are the only ones we tested.

I'm glad to hear of your positive experience with semver. Unfortunately, the issue with semver violations is that their impact has extremely high variance, easily ranging between "nobody noticed" and "ecosystem-wide pain":

- Most of the semver violations we found appear to have never been reported or noticed. After talking to a representative subset of the maintainers of those crates, they were generally surprised to find out a semver violation had happened.

- At the same time, a quick search for "violated semver" in Rust repos on GitHub will surface hundreds of issues, many of them heavily linked-to by issues in other projects. Every such issue represents dozens or hundreds of cumulative engineer-hours of triage and remediation in both the project where the semver accident happened and in affected downstream projects. All this work is regrettable, in the sense that both maintainers and downstream consumers would have preferred if it could have been avoided.

Most semver violations aren't intentional -- they are accidents that happen to all of us regardless of skill level, carefulness, experience, etc. Most of the time, those accidents go unnoticed or have minimal impact. Every so often, they trigger nearly ecosystem-wide consequences for a few days or weeks. (Some examples on my blog: https://predr.ag/blog/toward-fearless-cargo-update/ and https://predr.ag/tags/semver ) And sometimes, breaking semver on purpose can be the correct thing to do!

cargo-semver-checks aims to be a tool such that maintainers never have to say "if I had known this API change had happened, I wouldn't have published this version." Its biggest impact should be at minimizing or altogether preventing those ecosystem-wide semver issues that happen from time to time. Our day-to-day experience with semver should remain unchanged otherwise.

obi1kenobi··on Cargo-semver-checks: Scan your Rust crate for semver violations
cargo-breaking is cool! I only learned of it well after I had published cargo-semver-checks, but your logo is awesome and the tool was definitely a good idea, obviously.

If you ever feel like doing more work in the space, I'd love to work together on cargo-semver-checks!

obi1kenobi··on Cargo-semver-checks: Scan your Rust crate for semver violations
xD
obi1kenobi··on Cargo-semver-checks: Scan your Rust crate for semver violations
Author here. The same overall approach should work for Python and most other languages as well!

If you or anyone else is interested in building such a tool, ping me and I'd be happy to help you get started.

obi1kenobi··on Cargo-semver-checks: Scan your Rust crate for semver violations
Thank you!
obi1kenobi··on Cargo-semver-checks: Scan your Rust crate for semver violations
Exactly right. Most accidental semver violations aren't noticed by anyone, let alone most people. But every so often, an accidental semver violation causes ecosystem-wide breakage and lots of pain for both maintainers and users.

The goal of this tool is to aid in preventing regrettable accidents like that, by informing maintainers about semver issues before they are published.

obi1kenobi··on Cargo-semver-checks: Scan your Rust crate for semver violations
cargo-semver-checks is in the process of being merged into cargo itself, which would make the default publishing process into essentially exactly what you describe!

Running `cargo publish` would check for semver issues and alert if it finds anything. Users would be able to override it, of course, but most often they'd probably accept the version change or revert the semver break if it wasn't intended.

obi1kenobi··on Cargo-semver-checks: Scan your Rust crate for semver violations
Nice summary! I'd also add that part of the public interface you're on the hook for is never explicitly written down in the source code, and is instead derived automatically by the compiler based on a set of rules. There are excellent reasons why this is the case, but as you could imagine this doesn't make semver any easier.

For Rust folks, I'm talking about auto traits like Send and Sync.

More info and examples here:

https://predr.ag/blog/toward-fearless-cargo-update/

obi1kenobi··on Cargo-semver-checks: Scan your Rust crate for semver violations
Tool author here! I wrote a blog post that explains the motivation for building cargo-semver-checks here:

https://predr.ag/blog/toward-fearless-cargo-update/

TL;DR: Semver is particularly tricky in Rust and can be broken in many far-from-obvious ways. When a library breaks semver it can cause compile errors in every downstream bit of code that uses it. This sucks for everyone on all sides.

obi1kenobi··on Cargo-semver-checks: Scan your Rust crate for semver violations
Tool author here. I think the same overall approach I described here should work for Go as well:

https://news.ycombinator.com/item?id=36653954

One would just need a machine readable data source for Go package APIs, and would plug it into the Trustfall query engine. I'm not a Go expert but most languages I've worked with have something like that.

If you might be interested in building this for Go, ping me and I'd be happy to help you hit the ground running!

obi1kenobi··on Cargo-semver-checks: Scan your Rust crate for semver violations
No worries!

Trustfall is even more general than semver actually, one can use it to query any data source -- even HN itself:

https://play.predr.ag/hackernews

Here's a link to that project if you'd like to check it out:

https://github.com/obi1kenobi/trustfall

I gave a 10min conference talk on it last year titled "How to query (almost) everything"

https://www.hytradboi.com/2022/how-to-query-almost-everythin...

obi1kenobi··on Cargo-semver-checks: Scan your Rust crate for semver violations
cargo-semver-checks is the tooling you describe. It doesn't currently catch every possible semver violation, so it won't suggest version numbers for you yet, but that's the eventual goal!
obi1kenobi··on Cargo-semver-checks: Scan your Rust crate for semver violations
Tool author here. Your hunch is absolutely right. An analysis I and a few contributors did for Rust showed that over 1 in 6 of the top 1000 most downloaded Rust crates has broken semver at least once in a way our tool can detect & where we can produce a program that suffers a compile error due to the semver break.

This is obviously a lower bound, and yet it's quite a high number already.

I plan to write more about this on my blog in the coming weeks:

https://predr.ag/blog

In the meantime I put some more info (including a link to slides from a recent talk I gave on this) in another comment:

https://news.ycombinator.com/item?id=36653954

obi1kenobi··on Cargo-semver-checks: Scan your Rust crate for semver violations
cargo-semver-checks author here, thanks for the kind words :)

I've written several blog posts about how semver in Rust is particularly tricky and has unexpected edge cases, if you'd like to see specific examples of things non-obviously going wrong:

https://predr.ag/tags/semver/

Also, I've been working with the cargo team and the plan is to merge cargo-semver-checks into cargo itself, so it runs automatically on `cargo publish`. This would be similar to how cargo checks for uncommitted git changes: it will alert if it finds anything but you can override it if you think that's warranted.

obi1kenobi··on Cargo-semver-checks: Scan your Rust crate for semver violations
Author here! Excited to see this on the front page, ping me with any questions!

I recently gave a talk about the tool, addressing:

- why semver in Rust is important and harder than it looks

- how the tool works

- why it took until 2022-2023 to solve this problem

There's no recording but here are the slides: https://docs.google.com/presentation/d/1gVYM9-YLrBmRYSuHqW8r...

A lot of semver is just queries over APIs. For example: "find public functions that existed in the last version but don't exist in the new one." Any such function we find is a breaking change.

Cargo-semver-checks uses an engine designed to query any data sources called Trustfall, and uses rustdoc JSON (which has an unstable format that changes ~once a month).

Here's a query playground where you can try running queries over rustdoc:

https://play.predr.ag/rustdoc

More info about all this on my blog:

https://predr.ag/blog

obi1kenobi··on Ask HN: Could you share your personal blog here?
https://predr.ag/blog/

A few of my more popular posts:

- Falsehoods programmers believe about undefined behavior: https://predr.ag/blog/falsehoods-programmers-believe-about-u...

- To ace exams, get better at the easy questions: https://predr.ag/blog/to-ace-exams-get-better-at-the-easy-qu...

- Speeding up Rust semver-checking by over 2000x: https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov...

- Mediocrity can be a sign of excellence: https://predr.ag/blog/mediocrity-can-be-a-sign-of-excellence...

- Breaking semver in Rust by adding a private type, or by adding an import: https://predr.ag/blog/breaking-semver-in-rust-by-adding-priv...

- Debugging Safari: If at first you succeed, don't try again: https://predr.ag/blog/debugging-safari-if-at-first-you-succe...

← PreviousPage 2 of 7Next →