Comment by fngjdflmdflg | Hacker News Reader

Comment by fngjdflmdflg | Hacker News Reader

>Our method is similar to GANs in that a critic is jointly trained with the generator to minimize a divergence between the real and fake distributions, but differs in that our training does not play an adversarial game that may cause training instability, and our critic can fully leverage the weights of a pretrained diffusion model.

Very glad to see GANs (or GAN-likes) coming back! Also I don't know if the examples were cherrypicked but a lot of the images look more realistic than SD. For example the dog in the snow[0] is severely oversaturated in the SD version giving it that distinct AI look (sometimes referred to as AIslop). Also the landscape scene[1] has a random lens flare in the SD version which makes it look oversaturated as well. The MIT images are much better in this regard.

[0] https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_... vs https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_...

[1] https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_... vs https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_...

Can GANs prevent the ">10 fingers" problem?

The >10 fingers problem should be solved via higher quality data or conditioning rather than a better model architecture.

Are you suggesting AI generates images of people with more than 10 fingers because there are too many pictures of people with more than 10 fingers in the input data? That seems unlikely..

In some pictures not all 5 fingers on a hand are visible, so maybe it appears that humans have a variable amount of fingers?

Humans with less than 5 fingers per hand do indeed exist. How does that lead to a default of 7?

I meant more because people are holding objects or their hand is just at an angle where all fingers are not visible in the picture.

I don't understand how neutral networks operate, but my layman's guess is that when you sometimes see hands with 5 fingers visible, sometimes 4, sometimes 3, sometimes 2, sometimes 1, and sometimes 0, then it's not immediately apparent that it means that every hand has between 5 and 0 fingers.

Think of it this way, if the AI has ever only seen houses with a maximum of 10 windows in it's entire training set, is it so unthinkable that it sometimes draws a house with 12 windows? that's a "sensible" understanding about how houses have a variable amount of windows. It just doesn't work for fingers.

I'm sure the same issue would arise if humans had other body parts that came in large quantities, but almost everything else is either 1 or 2 like the nose or eyes.

Counting fingers is how humans do it, not necessarily how AI does. On a five-fingered hand, fingers are more likely to have neighbors than not. Why wouldn't the default be infinite fingers?

People holding hands, people high-fiving, etc.

Yes, but if AI cannot solve the fingers problem, then can it reliably generate images of things which we have relatively few example images of? We have an enormous amount of images of hands.

Higher quality could also mean the description attached to the original images in the dataset. If you were to describe a picture of someone you would never specifically call out the fact that they have 5 fingers per hand, we take it for granted, so that kind of information may never appear in the dataset.

But I think what the grant parent means by conditioning is non-textual conditioning, like ControlNet. This will always be more powerful than trying to describe something by text. Think about the description of a character in a novel vs the movie adaptation.

Outside of the work to study and improve available technologies ("how far can we push a hammer"),

you do not draw an individual with anomalous hands because you have an ontological model in which "humans normally have five fingers per hand".

Knowing "how the world works" is the appropriate source for subsequent expression of a representation.

Somewhere in the architecture a world model should be formed.

it's exactly the reverse: the issues with generative AI isn't the data anymore, but the models that are not able to understand the data

A lot of the problems of Stable Diffusion arise because of using a text encoder that isn't up to the task to encode the meaning sufficiently precise. For example, if you mention the word "green" in the prompt, it often carries over to the whole image instead of the object that is being described. Numbers suffer from a similar problem. Rendering text is compromised by tokenization.

One problem with hands might be that they are comparatively small. The model easily gets the big picture right (head, arms, legs), but hands are so-called high-frequency details and are additionally featured in lots of different positions, which are seldom sufficiently described in the captions of the training data.

>Very glad to see GANs (or GAN-likes) coming back! Adobe has had GigaGAN out a while now though?

Reply on news.ycombinator.com