Why shouldn't you be able to dismiss complaints about breaking changes with "the project uses SemVer"? That's the point of using semantic versioning in the first place.
If there are breaking changes, the major version is increased. Consumers upgrading to a new major version and not expecting breaking changes strikes me as irresponsible.