So, no, errors aren't special and should obey semver but you deliberately can (and should) signal that your users can't exhaustively match a thing you know will change.
However it's also only fair to note that the Rust ecosystem does not consider every [potential] breaking change a major one. https://predr.ag/blog/some-rust-breaking-changes-do-not-requ...
For this reason I personally don't like `#[non_exhaustive]`. I don't expect my code to keep compiling when I upgrade the minor version of my dependencies, and as long as the reason is simple, like a new enum variant was added, then I'm happy that it errored and will happily fix it.
This perfectionist attitude towards semver is why so many projects get stuck forever at v0. https://0ver.org/