There seem to be some nice developments into this direction recently? Something like the SfMNet (https://arxiv.org/abs/1704.07804) or the Correlation Filter Net (http://www.robots.ox.ac.uk/~luca/cfnet.html) for tracking.