You're mostly right, kind of. I'm going to
try and explain this, but it's not particularly easy. The confusion comes from both the nature of the "fair use" defense and the collision of legal vocabulary and common language.
Fair use is an affirmative defense, which means that even if a violation occurred in a strict sense, that violation is excusable. It's not an "I didn't do it" argument, it's an "I shouldn't be punished because..." argument. Self-defense is a textbook example of an affirmative defense -- strictly speaking, I would violate the law by intentionally injuring or killing someone, but I can argue that I was defending myself.
The question of an affirmative defense is not even relevant until there is a prima facie case showing violation, and often the fact-finder (jury or, in a bench trial, the judge) will actually find all the facts necessary for a judgement that infringement occurred before addressing the question of fair use, partly because the answers to many questions of fact for infringement are relevant to whether "fair use" holds.
In the end, you do end up with a judgement that no infringement occurred, but along the way, you often end up with findings of fact that do spell out a "violation".