The order of complexity difference between recognising figures on plain paper perpendicular to a scanning device in a controlled environment, and doing the same thing on huge amounts of non-standard chaotic data is why.
It's like a game of "Where's Waldo" on freaking crack.
You have literally no idea how complex this stuff is now then do you?
I do have an idea (literally, even) that there are additional problems having to do with extracting the house number images themselves from full-motion video, but that's an image registration problem and not an object recognition problem.