Hacker News Reader

Deep image captioning is essentially what you are describing.

Check out:

http://cs.stanford.edu/people/karpathy/deepimagesent/

http://cs.stanford.edu/people/karpathy/densecap/

Try looking for papers about image caption generation. For machine translation you can squeeze the information of a sentence into a vector (using a LSTM). You can then reconstruct the sentence in a different language. There was a paper how to generate such a "sentence representation vector" from an image instead, translating from "image" to "english".

Apart from the captioning work already mentioned, there's also visual analogy making work by Reed et al. http://www-personal.umich.edu/~reedscot/nips2015.pdf which is pretty exciting (sort of a converse to what you're proposing).