On puzzle #23 (id: 11e1fe23), I'm sure there's more than one possible valid answer from the examples given. You can't tell if the expected distance is from the gray square or from the RGB squares.
There are many examples where the test is slightly OOD (out of distribution), so the solver will have to generalize.