Patch for what though? Is the ability to use a patch taken into account because a patch to clang on Linux and a patch to appleclang on Mac have very different interpretations
Binary patches exist; there are fairly reasonable arguments that distributing binary patches (in a format that does not significantly contain the original work, e.g. bsdiff) is at least fair use, or possibly not a copyright problem at all.
Could you make the argument (not here) that binary patches are not a reasonable way of distributing a patch for a program that is frequently updated?