Hacker News Reader

Comment by BatFastard | Hacker News Reader

How can it possibly know what was in the white areas of the library? Is there a residual image?

Seems impossible that it guesses correctly.

It cannot know, but it can extend what’s around it to something that “looks believable”. I don’t fundamentally think it is different from asking an artist to complete the image. You’ll get an image, but no actual information about what was under the white.

It doesn't guess "correctly" at all. Zoom in on the image, and focus on the filled-in areas, they look really blurry. It just doesn't look very bad from a birds-eye view.

Probably this is a consequence of least-squares training; the 'average' solution is extremely unlikely.

Somewhat similar to content aware fill in Photoshop [0]. The untrained network can latch onto frequent patterns and match them to holes in the data.

Why doesn’t it paint everything white? Are these actually transparent images or are they somehow tagged?

[0]https://helpx.adobe.com/photoshop/using/content-aware-patch-...

Yes, for the inpainting, the parts to be painted (big white deleted areas) are supplied as masks, so it doesn't try to match them.

But its generating unique content in those areas...

Yes, every "run" of the network is generating pixels in those areas, but they're not being compared against the white (deleted) pixels. On the final run of the network, they're still not being compared against anything, except by us, visually, when we look at those pixels.

It seems like it should be similar in capability to a wavelet-based approach to e.g. image inpainting. In other words the neural network architecture essentially establishes a basis of features onto which the image is projected. Gabor wavelets are known to occur in the human visual system as the 'image elements' in a similar setup -- Gabor wavelets are kind of optimal, but clearly the features that this nn architecture uses are pretty effective too.

It doesn't it repeats/generates pixels that minimize E, so the behaviour heavily depends on the choice of E. for inpainting E is the difference between their generated x and the true x but they mask out the error in the white areas. The model gets good results by just copying nearby pixels.

because your brain runs on same CNN software i.e. doing pattern matching, not image matching.

Reply on news.ycombinator.com