1,728 karma · joined January 15, 2015
Your conclusion doesn't follow from your premise.
None of these models are trained to do their best on any kind of test. They're just trained to predict the next word. The fact that they do well at all on tests they haven't seen is miraculous, and demonstrates something very akin to reasoning. Imagine how they might do if you actually trained them or something like them to do well on tests, using something like RL.
Huh? I don't get this. The example right below seems a lot like coding with nodes to me. Perhaps they mean it's less procedural, more declarative? But that's the case with all of these visual programming interfaces. At least until you run into the limits of the declarative sub-language.
Sounds a bit like Noria: https://github.com/mit-pdos/noria
> Until now no neutrino produced at a particle collider has ever been directly detected. Colliders copiously produce both neutrinos and anti-neutrinos of all flavors, and they do so in a range of very high energies where neutrino interactions have not yet been observed. Nevertheless, collider neutrinos have escaped detection, because they interact extremely weakly, and the highest energy neutrinos, which have the largest probability of interacting, are predominantly produced in the forward region, parallel to the beam line. In 2021, the FASER collaboration identified the first collider neutrino candidates 13 using a 29 kg pilot detector, highlighting the potential of discovering collider neutrinos in LHC collisions.
Sqlite is supposedly high quality C code: https://github.com/smparkes/sqlite
For videos of someone (Casey Muratori) writing video game code and debugging it, Handmade Hero: https://handmadehero.org/
A blog post about how to write code by the same author: https://caseymuratori.com/blog_0015
For how to implement a fairly advanced type system, Typing Haskell in Haskell: https://gist.github.com/chrisdone/0075a16b32bfd4f62b7b
But, honestly, you're probably better off writing code yourself and learning by doing.
Expensive.
Generalization doesn't require learning representations outside of the training set. It requires learning reusable representations that compose in ways that enable solving unseen problems.
> On generalization - its still memorization
Not sure what you mean by this. This statement sounds self contradictory to me. Generalization requires abstraction / compression. Not sure if that's what you mean by memorization.
Overparameterized models are able to generalize (and tend to, when trained appropriately) because there are far more parameterizations that minimize loss by compressing knowledge than there are parameterizations that minimize loss without compression.
This is fairly easy to see. Imagine a dataset and model such that the model has barely enough capacity to learn the dataset without compression. The only degrees of freedom would be through changes in basis. In contrast, if the model uses compression, that would increase the degrees of freedom. The more compression, the more degrees of freedom, and the more parameterizations that would minimize the loss.
If stochastic gradient descent is sufficiently equally as likely to find any given compressed minimum as any given uncompressed minimum, then the fact that there are exponentially many more compressed minimums than uncompressed minimums means it will tend to find a compressed minimum.
Of course this is only a probabilistic argument, and doesn't guarantee compression / generalization. And in fact we know that there are ways to train a model such that it will not generalize, such as training for many epochs on a small dataset without augmentation.
There's no doubt that there are dimensions along which sugar and cocaine are different, but your conclusion is unsubstantiated, IMO. The paper's conclusion is much more restrained:
> Given the lack of evidence supporting it, we argue against a premature incorporation of sugar addiction into the scientific literature and public policy recommendations.
Lack of evidence is not necessarily evidence of lack. Most of the arguments the review makes are that a mechanistic link between sugar/fat consumption and addictice eating behavior has not been substantiated by the literature. The review does not argue that such a link is impossible or improbable, nor does it argue that addictive eating behavior is in a different stratosphere from addictive drug use behavior.
Sugar is both legal to possess and readily available at very low cost. No one needs to rob people to afford it.
Rats prefer both sucrose (table sugar) and saccharine (artificial sweetener) over cocaine: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1931610/
Metastable alloys (roughly, alloys that are only stable at room temperature because the atoms get frozen in place too quickly for them to reach the energetically favorable positions) are quite common in materials science, both for bulk metals and for semiconducting crystals. There are far more possible metastable alloys than stable alloys, so the range of material properties possible in metastable alloys is far wider than that in stable alloys. So, considering we have never discovered a room temperature superconductor, so far, it wouldn't be particularly surprising if it requires a metastable alloy.
Now we just need an iterative solver over both the structure and the "weights", and we get both architecture search and training at the same time
That's in the linked article about the ruliad, among other places in his writings about their physics project: https://writings.stephenwolfram.com/2021/11/the-concept-of-t...
Seriously though, I wonder what the ramifications to other parts of astrophysics would be if this is true.
What would you call "ultra-deep"? [1] shows how to train an RNN in a GPT-like mode, using parallel scan, with great performance on Path-X, which has a sequence length of 16k. It's based on prior papers doing the same thing but from a state space model perspective.
So is she going to be prosecuted for perjury?
Or put the modifier on the denominator so that the product and chain rules are obvious (the modifier only persists on the dx, not on the dy)
(300BT / 1.2BT) * 11 min * (1 hr / 60 min) = 45.8 hr
Still pretty incredible. That's an 18.8x speedup over 36 days.