You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. HOWEVER, NOTHING OTHER THAN THIS LICENSE GRANTS YOU PERMISSION TO PROPAGATE OR MODIFY ANY COVERED WORK. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so.
You're so focused on the first sentence that you missed the one I've put in caps. Once you link to the program, you've created a modified work, which as it says plainly right there in section 9, you cannot do without accepting the license, which is a contract that tells you what you can and can't do with that modified work.
I also suspect you're not really understanding that you don't have to touch the source code to the hypothetical AGPL'd JSON library to create a modified version of it. The whole theory of the (A)GPL is that you create a modified work by MERELY LINKING AND USING the library in your own code. The resulting program, including your own code that consumes the JSON library, is then a modified version of the library. This might seem counter-intuitive, but remember, the starting position legally is that you have NO RIGHT to use, modify, distribute, etc. a copyrighted work. You need a license from the owner. A license is a legal contract that can specify whatever terms and whatever conceptual framework it wants. You might think that linking to a library is merely "using" it, but the (A)GPL disagrees and that's the contract that tells you what you're allowed to do with it, take it or leave it.