1,050 karma · joined May 26, 2018
1/ For best journals: a non-replication betting market for peer-reviewed and published papers. Grants to replicate papers that are highest in the betting pool.
2/ Citation index were citing and publishing a paper that does not replicate lowers the score.
(This should be first used in machine learning research)
There are many others.
ML techniques will be used to cut that latent variable space in both quantum chemistry and molecular mechanics based methods.
If we could simulate and observe what happens with complex chemistry accurately, it would completely change the biology, medicine and materials science.
My current hypothesis is:
Users who click ads and generate ad revenue like this garbage. Google learns to serve this outlier group of users. Killing the experience for everyone else is just a side effect.
ABSTRACT
Psychologists have repeatedly shown that a single statistical factor—often called“generali ntelligence”—emerges from the correlations among people’s performance on a wide variety of cognitive tasks. But no one has systematically examined whether a similar kind of“collective intelligence”exists for groups of people. In two studies with 699 people, working in groups of two to five, we find converginge vidence of a general collective intelligence factor that explains a group’s performance on a wide varietyof tasks. This“cfactor”is not strongly correlated with the average or maximum individual intelligence of group members but is correlated with the average social sensitivity of group members, the equality in distribution of conversational turn-taking, and the proportion of females in the group.
I run some large ML models in my home PC and I get NaN's and some out of range floats every month or so. I have spent hours debugging but doing the same computation with the same random seeds does not recreate the problem.
How about GPU's and their GDDR SDRAM? Do they have parity bits?
From the paper they cite
algorithm kWh-PUE
---------- -------
ELMo 275
BERT(base) 1,507
NAS 656,347
But the actual learning process of a single brain uses very little training examples and is very energy efficient. A brain that uses only 20 watts of power.
https://github.com/ml-jku/hopfield-layers
https://old.reddit.com/r/MachineLearning/comments/i4ko0u/r_h...
Neural doohickeys find real uses in computational chemistry. Accurate ab initio solvers (many-body Schrödinger equation) are very slow and everything else is just heuristic-, quasi- approximation mixed with semi-empirical tricks.
When someone makes even marginally better heuristic improvement, it's more likely that they sell it or start a company than post paper in arxiv. There is serious money to be made.
Yes. Humans can learn to answer questions even when questions (test cases) don't come from the same distribution as the training set.
You need to have some amount of analytical capabilities. Analytic as the ability to form distinct and modular concepts from the input and form representations that can be composed together to generate instances that don't exist in the distribution. The low-level example of this ability could be blind source separation.
Typically the ability to learn << ability to represent in most cases. Universal approximation theorem type results don't say anything about learnability. GPT-3's abilities quickly fade once it gets outside the distribution it was trained with.
You are correct. My main argument was that in-distribution learning is not enough. You can's fix that problem with more data as many responses to my comment seem to assume.
I think out-distribution leaning and small data requirement are connected. If agent can understand the concept separate from the chain leading from sensory inputs, it can understand what it's doing in novel situation even without examples.
Our genes give us priors to evade lions and pick berries. Humanity as cultural organism can reprogram us to more abstract tasks, but it takes easily 10-20 years of training and we are relatively inefficient in them compared to our original task.
But even then. I don't think current in-distribution learning is enough even if we scale it up. There must be something else. We are at least 3-5 Turing awards away from AGI, I think.
16 year old human has experienced only about 400 million wakeful seconds.
The largest GPT-3 model "GPT-3 175B" is trained with 499 Billion tokens (one token is maybe equal to 4 characters in text).
Human reading/talking/listening equivalent of 200 pages of text per day for 80 years would be just 13GB of raw data or 3B tokens. You could also make an estimate using 39 bits/s as the normal information rate humans can absorb and get the same order of magnitude estimate.
It's not completely wrong to say that we are figuring out how far interpolation from data can go as a learning method. Even the most advanced deep learning is in-distribution learning and system 1 thinking (Kahneman's term). Just run through data until the model can interpolate accurately between data points.
We must figure out how do learn models that allow out-of-distribution learning or these models fail the Turing test after a few questions.
For working memory and visual memory the number 4 pops up again and again.
When you are holding 5-9 things in memory you can't 'work with them' effectively. You might be chunking objects together or alternating very fast. Verbal memory can be tricky that way. (computer analogy: 4 is number or registers, 3-5 other objects may be stored in L1 cache).
Good starting point: Visual short term memory http://www.scholarpedia.org/article/Visual_short_term_memory
[1]: Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185. doi:10.1017/S0140525X01003922
[2]: Cowan, N., Elliott, E. M., Saults, J. S., Morey, C. C., Mattox, S., Ismajatulina, A., et al. (2005). On the capacity of attention: Its estimation and its role in working memory and cognitive aptitudes. Cognitive Psychology, 51, 42-100. doi: 10.1016/j.cogpsych.2004.12.001
[3]: The Magic Number 4 ± 0: A New Look at Visual Numerosity Judgements (1976) doi: 10.1068/p050327
[4]: The Magical Mystery Four: How is Working Memory Capacity Limited, and Why? (2010) doi: 10.1177/0963721409359277
- 1.230 Gb/year
- 24 Gb in 20 years = 3 GB
3 GB is theoretical upper limit and overestimate to what human can learn verbally until they are 20 years old. It would require being awake and communicating 24/hours per day.
Of course, we don't learn all what we hear and much of it is repetitive. It's just raw data we could receive and communicate verbally in theory.
As uncompressed English text stream that would be probably 15-20 GB. Imagine Deep Learning model that requires only that amount of training. Not not as training corpus that is gone trough again and again hundreds or thousands of times, but as total input when communicating with the external world.
For AI researcher Game AI is like porn. It's cheap tricks and obviously fake but oddly fascinating. Sometimes you find a new trick you want to try in real life.
Marvin Minsky once said "I bet the human brain is a kludge." If I had to bet, I would say that human brain is full of dirty tricks, incomplete solutions, shortcuts and artificially limited problem spaces evolved to pick berries and avoid tigers, not to understand the world. Combining many tricks together can create illusion of generality that is very convincing.
I think you could do this by changing the value function:
- reward short game over long games,
- draw reward closer to loss reward,
- reward capture more.
In fact, I would like to play against AlphaImmortal specially trained to play against my Elo-rating level opponents with 51% win rate. That would be fun!
Even basic research is can be sensitive to omitted parameters, setups and starting conditions. Honest mistakes, accidental omissions and failing to spot sensitivity to parameters happens all the time. Writing "A clear explanation of any assumptions" is rarely comprehensive. Its easy to miss things and become blind to some fine details.
Discovering and documenting of a new interesting phenomenon and dynamics is also part of basic research. Publishing experimental discoveries without explanation should be always reproducible.
Particles (individual humans or small groups) move around in problem space as particles with position and velocity. Each individual's movement is influenced by its best known position locally but also the the best known global positions in the search-space. When better positions are found by others, individual change their course (take hints) and move towards them.
The problems would be the same. Too much randomness and it's just a random search. Too much convergence leads to local optimums.
Science vs applications would be actually a good way to divide NeurIPS into two conferences to get focus back: NeurIPS and Applied NeurIPS. One wold present science, the other tweaks, tricks and methods to solve everyday problems.
Papers like "Partially Encrypted Deep Learning using Functional Encryption", "Dancing to Music", "Differentiable Cloth Simulation for Inverse Problems" would go go the engineering conference for the "most people".
Is this brain cell your ‘mind’s eye’? https://blog.frontiersin.org/2019/10/01/systems-neuroscience...
Neuroscience is full of problems of the hardest kind.
AGI is a scientific problem of the hardest kind, not an engineering problem where you just use existing knowledge to build better and better things.
Marving Minsky once said that in mathematics just five axioms is enough to provide the amount of complexity that overwhelms the best minds for centuries. AGI could be messy practical problem that depends on 10 or 25 fundamental 'axioms' that work together to produce general intelligence. "I bet the human brain is a kludge." - Marvin Minsky
The idea that if many people think this problem very hard, the problem is solved in our lifetime is prevalent. It's not true in math and physics, so why would AI be any different? Progress is made but you can't know if there is breakthrough tomorrow or if it happens 100 years from now. Just adding more computational capability is not going to solve AI.
Currently it's the engineering applications and the use of the science what is exploding and getting funded. In fact, I think some of the best brains are lured from the fundamental research into applied science with high pay and resources. What the current state of the art can do now has not been utilized fully in the economy and this brings in the investments and momentum.