If you see convolution as basically truncated recurrence this approach ties in very strongly to recent approaches to machine translation using recurrent nets. I guess depth should allow you to find longterm dependencies, but the fact that CNN were designed for images which have strong local structure and much weaker long term structure makes me think RNNs are better for language, where we see a lot of important long term dependencies. As an example: "The man with the long brown hair entered the saloon" - I would tie saloon and man as the key pieces of that sentence, but that dependency is pretty long and somewhat different than natural images where you don't really expect the corners of images to have any strong relationship in general.