This could be highly interesting. Especially when starting with the FFT butterfly structure for connections and the corresponding weights as network initialization and then adapting that to real data via training.
This is a great idea, I bet one could find a bit-faster fftw that would be close enough for many applications, but it would apply a statistical approach to each weight.