> There is no mechanism to say "oh these neurons will never activate, let's prune them"
Not at training time. But after training you can evaluate which activations rarely fire and simply remove their columns from the weight matrix.
Or you could re-randomize them and continue training.