https://arxiv.org/abs/1311.2901
Check out figure 2. The network learns composition from fundamental shapes and gradients to compositional ones. It's kinda awe inspiring.
Not NN specific, but some more work on explanations: https://homes.cs.washington.edu/~marcotcr/blog/lime/
Visualization of attention mechanisms is pretty cool for explanations: https://arxiv.org/pdf/1502.03044.pdf http://torch.ch/blog/2015/09/21/rmva.html