MAXPOOL

You don't need to memorize rules when studying math. Just like you don't need to spend any time to memorize syntax for programming languages. You automatically remember things you use a lot.

Once you have spent countless hours doing exercises to the extent that you understand the math, you already remember the rules. If you have not spent countless hours doing exercises, you don't understand anything at this level.

You don't hire a programmer who has read all the books and 'understands' programming but has never programmed. It's the same with math. You don't just read a math book from start to finish. You can use wolfram alpha for visualizing functions, not for learning math.

Capsule networks are attempt to get past the limits of current approach to deep learning. It took 20 years until the the current paradigm in deep learning started to generate results. You start with toy problems and explore.

Hinton's "What is wrong with convolution neural nets?" is still true to the state of the art in convnets. Gradually turning features into objects like we do in deep learning is limiting. Distinguishing feature integration from object detection like humans do and how to do it is an open question. It can take another 10-15 years.

Learning multilayer convolutional representations of statistical features is roughly equal to taking few first few layers in visual cortex and stacking them. Creating higher and higher stacks is not going to solve vision.

We are essentially building a frog with better and better visual perception in the hope that it could become a taxi driver. It will become a totally amazing super-frog with super-vision, but it's still just a frog with frog-like visual perception and limits. Using pre-attentive feature recognition stage equivalent for complex object recognition can fake human like object recognition when we force it, but it's wrong approach. We get these catastrophic failures because we hit the limits.

Features seem to exist independently from one another in the early processing stages of human perception. They are not associated with a specific object either. Human perception is not gradually turning features into objects like we do in deep learning. Properly distinguishing feature integration from detection and how to do it is a open question.

Did he really make this claim? I didn't listen the podcast.

If true, this sounds completely unrealistic especially with the way Tesla is approaching self-driving cars. It's possible that Musk will showcase something within year, but I'm certain will not be a full self-driving car in any meaningful sense.

If there is betting market with well defined bet terms, I could bet some money on it.

I found this incredible book draft because Scott Aaronsons glowing praise of Avi Wigderson made me curious https://www.scottaaronson.com/blog/?p=4156

>Congrats to Avi Wigderson for winning the Knuth Prize. When I was asked to write a supporting nomination letter, my first suggestion was to submit a blank sheet of paper—since for anyone in theoretical computer science, there’s nothing that needs to be said about why Avi should win any awards we have. I hope Avi remains a guiding light of our community for many years to com

Avi Wigderson homepage is full of interesting stuff. https://www.math.ias.edu/avi/

In the 19th century William Whewell developed concept of "Consilience of Inductions". Evidence from independent, unrelated sources can "converge" on strong conclusions. We should feel more confident about our conclusions when we have lots of lines of evidence that stand alone independently, preferably collected by different people, in different parts of the globe or at least different universities. Maybe they don’t even like each other.

Many contrarian views grow long time in small small minded groups or even withing one individual. They have high internal consistency and logic. When exposed to outsiders, they usually spot some obvious errors immediately.

Open Review https://openreview.net/forum?id=SkfMWhAqYQ

The human retina communicates to brain with 10 Mbps. The 4-year old's brain has experienced 9 months in a womb and 4 years outside the womb. That's less than 150 million seconds. More than third of that while sleeping.

Assuming the full sensory data rate is 50 Mbps total (probably less), 4-year old has received 100×10^6 × 50Mbps = 625 TB sensory information while in a waking state.

"for all intents and purposes"

When you get brain damage that changes your behavior, it's for all intents and purposes 100% environment.

Almost always both.

Environment triggers changes in gene expression and reverse is also true. There are multiple ways of how nature and nurture interact.

Take for example stress during childhood. It seems that stressful environment leads epigenetic changes an behavioral changes like lower impulse control, aggressive or more promiscuous behavior.

Is the change in behavior damage caused by the environment, or is it evolutionary adaptation to uncertain and dangerous environment? It's possible that dangerous environment triggers different epigenetic survival strategies? Breed faster, be more aggressive and do quick decisions.

Differentially Methylated Genes in Saliva are linked to Childhood Stress https://www.nature.com/articles/s41598-018-29107-0

Not my exact field, but I keep track of the research for potential applications.

The general vibe is that there are potentially large welfare gains to be achieved if algorithms, ML or statistical methods are integrated into human decision making in principled way. People have innate tendency treat noise as a signal. That does not mean that the dangers mentioned in the article are not real [2]. We should be very aware of them in their current forms and prevent repeating them.

Few of my favorite papers:

1. Human Decisions and Machine Predictions The Quarterly Journal of Economics, Volume 133, Issue 1, 1 February 2018, Pages 237–293, https://doi.org/10.1093/qje/qjx032 https://www.nber.org/papers/w23180

2. Dissecting Racial Bias in an Algorithm that Guides Health Decisions for 70 Million People (2019) https://dl.acm.org/citation.cfm?doid=3287560.3287593

3. Simplicity Creates Inequity: Implications for Fairness, Stereotypes, and Interpretability https://arxiv.org/abs/1809.04578

4. Direct Uncertainty Prediction for Medical Second Opinions https://arxiv.org/abs/1807.01771

ReLU is very simple in this regard. In the plain form it's just affine transformation followed by 'a viewport'. The mapping trough multiple layers is alternating affine transformations and windows into data. Learning is combination of squeezing and rotating data so that it can be seen or unseen from the window and rotating window frames to do the same.

Any extra stuff, like batch normalization between the layers can again introduce more complex nonlinearity.

They are not hyped. Capsule nets are in early phase of research.

The bleeding edge deep learning research that tries to push the science forward is trying to develop new ideas. Hinton et. al developed the current deep learning revolution gradually over decades. Next non-incremental revolutions in the field may take similar time to mature.

> The cerebellum had been far too neglected in AI research.

Cerebellar Model Articulation Controller (CMAC) https://en.wikipedia.org/wiki/Cerebellar_model_articulation_...

A Historical Review of Forty Years of Research on CMAC https://arxiv.org/pdf/1702.02277.pdf

It's possible to infer causation from correlation without experiments if you add some general assumptions.

One trick in causal discovery is additive noise. If X and Y are noisy correlating variables and X is causing Y, assumption that the noise in X is present in Y but not vice versa may reveal the direction of the causal arrow.

Causal Discovery with Continuous Additive Noise Models http://jmlr.org/papers/volume15/peters14a/peters14a.pdf

Nonlinear causal discovery with additive noise models https://papers.nips.cc/paper/3548-nonlinear-causal-discovery...

Humans seem to have causal reasoning ability that is very ad hoc. It works well in practice but it's not principled. There is not enough time to do experiments to establish facts. Correlation is causality seems to be a good heuristics.

I think that that AI will eventually learn to build causal models in the same way. Build a quick and dirty causal models with unfounded assumptions and see what works. Hold multiple effective conflicting causal theories that apply in different situations without any consistent model.

It's deep and large model. Maybe unnecessarily so.

In image processing a single pixel or region almost always belongs to one object. Car can be behind a tree, or tree can be behind a car but they are not transparent and mixed together.

In raw 1d radio, audio, ECG or EEG signals there can be multiple additively overlapping singnals across different frequency ranges. Signal patterns are transparent to other signals. ConvNet on raw singnal must learn to discriminate overlapping pattern combinations.

You still may want to replace softmax layer with support vector machine for classification sometimes.

The problem is not the data. The problem is the need for high quality data. Current ML is data driven statistical learning. ML tries to learn a model that describes the distribution. It's impossible to get similar performance as the best reference implementation (human brain) using this approach. https://i.redd.it/kvvgv6zzhtp11.png

Think of 16 year old human:

* it has received less than 400 million wakeful seconds of data + 100 millions seconds of sleep,

* it has made only few million high level cognitive decisions where feedback is important and delay is tens of second or several minutes (say few thousand per day). From just few million samples it has learned to behave in the society like a human and do human things.

* Assuming 50 ms learning rate at average, at the lowest level there is at most 10 billion iterations per neuron (Short-term synaptic plasticity acts on a timescale of tens of milliseconds to a few minutes.)

Humans generate very detailed model of their environment with very little data and even less feedback. They can learn complex concept from one example. For example you need only one example of pickpocket to understand the whole concept.

I'm just eyeballing it, but it seems like linear trend line of the nowcast (blue line), would be very close to the official forecast (red dot).

Yes.

Deep learning is not mimicking how our brains work. It's just brain inspired. It may mimic of how some primitive areas of brain work in very low level in very crude way.

Take for example machine vision. ConvNET is inspired by the lowest levels of human visual cortex and there are analogies to be drawn, but no direct mimicry.

> important results are simply bugs

Probably none. If the paper is important and collects citations, the algorithm is in use. Computer science != working code.

Code is required when you produce something where the scientific importance is less clear. There is need to provide more evidence. Many papers are just "Hey I made some some tweaks and it works in this particular case." Those papers should have working code.

If you have a good paper with important result, providing code and data is not necessary.

Providing the code to replicate is good form. It shows good faith and confidence. Exact replication (exactly replicating the study) is just the starting point to check that the code works and no obvious mistakes were made.

replication / reproducibility / hyperparameter sensitivity

If the research yields something really important and the method is well documented, usually it can be easily checked without having the data and the code. Things like dropout, batch normalization, residual learning, .. work over multiple different datasets and hyperparameters. You can reproduce the results without faithfully replicating the experiment.

If the claimed result vanishes unless you have the exact data, code or the hyperparameters, the research can't be said to be meaningfully reproducible in the scientific sense. Hyperparameter sensitivity is ML equivalent to P-Hacking.

You would think that computer science can't have replication failures, but it can. And I'm talking about my own field: machine learning. There is so much hype that I suspect that people are pushing papers and trying to actively hide the irrelevance of the methods and algorithms they develop.

Artificial intelligence faces reproducibility crisis http://science.sciencemag.org/content/359/6377/725

Reproducibility in Machine Learning-Based Studies: An Example of Text Mining https://openreview.net/pdf?id=By4l2PbQ-

Missing data hinder replication of artificial intelligence studies http://www.sciencemag.org/news/2018/02/missing-data-hinder-r...

>In a survey of 400 artificial intelligence papers presented at major conferences, just 6% included code for the papers' algorithms. Some 30% included test data, whereas 54% included pseudocode, a limited summary of an algorithm.

The way to tackle the problem you state would be finding similar bounds with regularization.

This is not controversial statement. Most Deep Learning pioneers (like Hinton and Bengio) agree and try to find a new way out.

Of course there has been many breakthroughs. Solving exploding or vanishing gradient problem is important. Throwing in more hardware can't solve the problem if gradients don't work.

The large picture is that they are are all just technical tweaks for the same basic underlying idea that has existed since 80's and 90's. Just adding more layers is not enough.

Deep learning is still statistical learning. DL algorithms attempts to learn output distribution that matches the target distribution.

Only 100 of those are sensory neurons.

Only two neurons are for memory (interneurons as the neural substrates of mechanosensory memory).

Sean Carrol has a interesting podcast episode (and transcript) related to this type of research for those who are interested.

Episode 16: Coleen Murphy on Aging, Biology, and the Future: https://www.preposterousuniverse.com/podcast/2018/10/01/epis...

> Aging — everybody does it, very few people actually do something about it. Coleen Murphy is an exception. In her laboratory at Princeton, she and her team study aging in the famous C. Elegans roundworm, with an eye to extending its lifespan as well as figuring out exactly what processes take place when we age. In this episode we contemplate what scientists have learned about aging, and the prospects for ameliorating its effects — or curing it altogether? — even in human beings.

Yet another way to visually understand complex functions is to think them as 2D plane to 2D plane transformations. You draw a picture, grid or curves into complex plane, then run it trough complex function you are interested in and see how it looks like.

Here is w = e^z:

https://i.imgur.com/pAALOh2.png

Just look at the above picture and every detail until it starts to makes sense.