Understanding the role of individual units in a deep neural network

Perturbing units and observing change is so much like the early days of neurology and neuroscience. Lesion studies in post-surgery patients or those with stroke or brain damage told us a lot, removing part of the circuit and observing what changed.

We also think there should be more Ablation Studies in deep learning research. To make it easier, we are working on automating the process of disabling components of models with a framework called Maggy. Pass in your Keras Model and specify which layes/components to mask out and we will generate all models and train them in parallel. https://databricks.com/session_eu20/parallel-ablation-studie... https://github.com/logicalclocks/maggy

This is all super cool, but for those of you who tl:dr;ed, check out their application - creating a "concept painting" app. You can literally click "door" and spray on the picture, and it'll put a door there for you.

http://gandissect.csail.mit.edu

http://gandissect.res.ibm.com/ganpaint.html?project=churchou...

The most interesting use cases for this would be applying it to a game playing AI. We might be able to find novel strategies to games.

Also interesting to find out how the network that solved protein folding works.