Where I work, we call that a breaking change. If a consumer of our API codes for a particular response, and we change that response, then we could break their application.
No one wants to have to validate deeply nested objects to test if they do indeed return the thing we promised to return in our contract...
The difference with bugfixes is that pre-bugfix the intended API contract - including behaviour, not just signature - was not respected by the implementation and post-bugfix the implementation becomes (more) correct WRT the intended contract.
IOW:
- On a semver patch bump the API contract does not change but the implementation of that contract was incorrect.
- On a semver minor bump the API contract receives an addendum, the previous contract is still in effect as a subset of the new contract.
- On a semver major bump the preexisting API contract is voided and is replaced by a new contract. The new contract may or may not include all or part of the old contract as a subset. Breaking change => major bump is an implication, not an equivalence: one is completely free to bump major purely for marketing reasons.
Of course, the difference between theory and practice is that they're the same in theory: the contract is an ideal, the implementation may come close but will always fall short in some way.
I feel like many of both semver detractors and advocates get lost in theoreticals, reality is fuzzy, just use the damn thing. Or don't and pick up another scheme, but be abundantly clear about how your versioning scheme works so that users can make sense of it and automate version requirement declarations in their dependency manager.
I think the confusion over what constitutes a breaking change stems from the fact that, in practice, contracts are often very underspecified, sometimes to the point of being omitted entirely ("This function just does the obviously correct thing"). A classic example would be whether an API that munges filesystem paths should resolve symlinks or not -- if this is unspecified, then changing that behaviour is technically not a breaking change, but is likely to upset some fraction of your users, who may have even done some experiments to find out what the code does, and assumed that that undocumented behaviour won't change.
(This is not an argument against semver at all -- just an observation that contracts are hard, and mostly need to be much more detailed than they typically are, and this "detail gap" seems unlikely to ever completely go away.)
Another example would be Ruby 2.4 changing String#upcase & friends from handling ASCII only to being unicode aware with a new argument to override and changing the default. Technically a breaking change but actually silently fixes a ton of bugs.