After thinking about it for a while I understand it now. The inverse-square law applies to a point source. As you move further away the intensity drops off for each point, but the points get closer together in your field of view to compensate.
One weird thing is that many of the stars in the sky would be as bright or brighter than the Sun, but their "disc" is just so tiny that we perceive it as being dim.
The inverse square law always applies, but it relates to the total amount of light received by a constant area observer (or density at a pointlike observer). Once you're projecting the image of the source onto a plane with a lens, that doesn't map to surface brightness.
Another place where the inverse square law breaks down is with beams of light, in the near field - consider something like a laser beam. While it will expand, and eventually follow the law once you get far enough, it very much doesn't at close range. And there's an interesting parallel here: to an observer, a collimated laser beam looks like a perfect pointlike object at infinity (modulo the diffraction limit, of course). Another way to see it is that focusing a beam of light in a certain way creates a virtual emitter much further away, and so you need to add that to the distance you plug into the inverse square law.