Without #[non_exhaustive], adding a variant is always a semver breaking change.
With #[non_exhaustive], adding a variant is not a breaking change, but only if that variant can safely be ignored. If it can't be, then it's still a breaking change, BUT you don't get a compiler error. That's a very big drawback IMO.
This makes me think of how the PNG format handles this. For context, PNG files consist of a list of chunks of different types (image data, color profile, text comment, etc). Each chunk has some flag bits that tell you whether it can be safely ignored when decoding the image, safely retained when the image data is changed, etc. That way the decoder can ignore unknown ignorable chunks and error on unknown non-ignorable ones.