Reservoir computing
en.wikipedia.org
en.wikipedia.org
[1] https://link.springer.com/chapter/10.1007%2F978-3-540-39432-...
Also the Liquid State Machine paper they cited, "Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations": http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.5.8...
Fascinating idea. If I understand correctly, it's like an extreme version of data pre-processing. If you're trying to figure out whether an audio clip is saying "zero" or "one", analysing the raw amplitude data is pretty tough going. Instead you could run it through a Fourier transform in the hope that the clip's frequency content would be easier to analyse. If that doesn't help, maybe a wavelet transform, or something more fun like, uh, the "inverse Fourier transform of the logarithm of the squared magnitude of the Fourier transform"
In a sense, it doesn't really matter what pre-processing you do, provided that the differences between "zero" and "one" are more distinct in the output than the input. This is the "separation property" that the papers mention: important differences get magnified at the expense of unimportant ones. If that's true, your final analysis will have a lot less work to do.
What's cool about this is that "anything that magnifies important differences" is a pretty open-ended requirement, leaving you free to choose pre-processing that's easy to implement in hardware. In this case, fluid dynamics has the desired properties, and the laws of our universe make it very easy to implement a fluid simulation using actual fluid in an actual bucket.
Perhaps there are other systems with similar separation properties that are even easier to implement in hardware. Maybe something with electromagnetic waves, like in time-domain reflectometry? Even if such a system's behaviour is uninterpretable to us, it might still provide useful pre-processing to a machine learning algorithm.
[1] http://www.physnews.com/nano-physics-news/cluster1837307157/
First, let me attempt a summary in language that makes sense to me. You want to interpolate a function f:R^n to R^m. So you pick a "random non-linear dynamical system", and compose f with the output of this. To be concrete, let's say you pick a polynomial map F: R^n to R^n of degree at most k for some fixed k with random Gaussian coefficients. Let g(x) = h(1), where h' = F(h) and h(0)=x. Now, given training data (x_i, y_i), let z_i = g(x_i), and interpolate the function f* on the training data (z_i, y_i) using whatever method you like (least squares or a Neural net or whatever). Given test data x, guess f(x) = f*(g(x)).
Is this correct? Why is this plausibly a good idea? Why have the function g be the output of a dynamical system, and not a random matrix? Or if you want g to be nonlinear for some reason (what reason?), the output of a random polynomial or a random Fourier series? What happens if two clusters in the initial data which are well separated and map to different well-separated outputs in the codomain get mixed after composing with the dynamical system map g? Or what if g is not even one-to-one?
Sort of acting like a hash function - small differences in the source become large differences in the output.
They're expensive to train, in computing time and data-requirements, but they're the closest thing we have to trained programs.
Reservoir computing approaches usually have the intermediate layers be recurrent, eg they implement a dynamical system. Theoretically, this is actually Turing complete, although good luck programming it. But in any case, the range of behaviors — transformations in the data — that a dynamical system can implement is much more powerful than just a basis expansion. However, whether this is what is happening in actual practice is really, really unclear to me. A lot of recurrent neural networks aren't doing anything more than an equivalent feedforward network, and the same thing may be true here: reservoir approaches might, for the most part, really just be performing nonlinear projection to higher dimensional spaces, and then the output layer is being trained to classify those patterns.
Seems like a decent introduction (not versed in the topic, but this is a more informative article than the wikipedia page).
Check out Figure 1 for a model like what krastanov discusses.