Phase I is segmenting the image into rectangular regions which contain an "object". That's done using the approach in this paper.
http://koen.me/research/pub/uijlings-ijcv2013-draft.pdf
The regions can overlap, and one region can be completely contained in another. Regions found may turn out to be useless, so the goal is to find too many, rather than too few. That part is reasonably straightforward, and you can see how well it's working visually.
Phase II is described in this paper:
http://arxiv.org/pdf/1311.2524v5.pdf
Each rectangular region is rescaled into a 277 x 277 pixel square.
Then they apply a convolutional neural network to each square:
"We extract a 4096-dimensional feature vector from each region proposal using the Caffe implementation of the CNN described by Krizhevsky et al. Features are computed by forward propagating a mean-subtracted RGB image through five convolutional layers and two fully connected layers."
So now they have a 4K vector of values for each region. I thought CNNs had to be trained, but what would you train this against? The 4K vector has no explicit meaning yet.
Phase III, in the same paper, uses a manually tagged training set and a support vector machine to try to train a model of those 4K feature vectors. The output of this is a set of tags for each region, each with a probability. This seems to be a straightforward machine learning system.
The new Stanford paper takes that data and, using training data from captioned images, turns those tags into sentences. (I'm more interested in the object recognition part than the natural language part, so it's the upstream steps I'm trying to understand.)
I'm amazed that this works, and have no clue how one debugs such a thing. None of this is all that big in terms of lines of code, but it's not at all clear what will work.
There's been a lot of progress in the last few years.