Hacker News Reader

noob here. how does it know that the word description embedding matches with the image embedding? how does it know the color red is in that image?

The model is trained by giving pairs of examples. So that it learns to generate an internal representation of images, texts or whatever inputs.

There are other ways to train, but this is the basic example.