Correct me if im wrong, this is just a question originating from my own curiosity, but could you not apply the same or similar techniques to classical NN architectures? I dont see exactly how the differentiability of the architecture matters? I.e. in between epochs randomly perturb weight values (similar effect I understand to introducing scheduled noise or non-isomorphic transforms on the features), implement a scheduled causal drop in/out scheme etc? I know of people that train models these ways as a form of regularisation and they claim better fits too.

Enjoy your adventures in the Dark Forest.