One major, major advantage that medical imaging has for deep learning is the similarity of each data point, especially the 'background data.' For instance, human brains typically look very similar across individuals (up to scanning parameter differences), except in the abnormalities - which are often precisely what you want to highlight.
As an example, I recently trained a neural neural network to perform a useful task for our lab using 3 (!) hand-labeled brains.