This is fantastic. I wonder if we've passed a tipping point where some of these fundamental machine learning algorithms will be taught the same way we learned the C++ STL back in the day. Really something like a spam filter should look like a shell script, where you pipe things around and use off the shelf tools, rather than spending inordinate amounts of time trying to implement building blocks (imaging having to write grep before you could even start).

Just out of curiosity, does anyone have a link on NN training parallelization? I’m just hitting paywalls. I’m curious if it’s simple or if there are dependencies between neurons that make it hard. As I see it, the next big hurdles after ubiquitization are distributed training (possibly with GPUs) and empirical data on the best tuning parameters for various problem spaces (which will come largely for free when training happens 5 or 10 orders of magnitude faster than today). Eventually we’ll have an answer as to whether AGI can be composed of basic building blocks or if we’re still missing some fundamental insight.

For example, genetic algorithms are fairly easy to parallelize because by their nature they use large and independent populations. About the time I was graduating college in the late 90s, there was great interest in using them to find the initial weights in NNs to avoid local minima, but then I didn’t hear much about it since. Have there been any breakthroughs?