I think what he means is that in signal processing you can define the signal analytically and perform the Fourier transform analytically, without sampling. The Nyquist rate only comes in when you talk about digital signal processing.
This was what I meant initially, but I realized in a lot of NNs that it might be pretty difficult if not impossible to define the signal analytically.
Ah, if I take your meaning of analytically correctly, I think you are basically arguing for symbolic reasoning.
I do still see the appeal of that. Seems solidly not a promising path, at this point. :(