So which papers are you talking about?
56 karma · joined June 7, 2020
You haven't even read the paper, have you? Otherwise you'd see that it's Hinton and Bengio who are cited far more often than they should be. Just look at disputes B1, B2, B5, H2, H4, and H5 to see how they republished parts of his work again and again without citing it. No honest scientist can approve of something like that.
> "the inventor of an important method should get credit for inventing it. She may not always be the one who popularizes it. Then the popularizer should get credit for popularizing it (but not for inventing it)." Nothing more or less than the standard elementary principles of scientific credit assignment.[T22] LBH, however, apparently aren't satisfied with credit for popularising the inventions of others; they also want the inventor's credit.[LEC]
I think there is a reason why the ACM Turing awardees have never tried to defend themselves by presenting facts to the contrary: because they can't.
This might get interesting:
> The "Policy for Honors Conferred by ACM"[ACM23] mentions that ACM "retains the right to revoke an Honor previously granted if ACM determines that it is in the best interests of the field to do so." So I ask ACM to evaluate the presented evidence and decide about further actions.
Don't you know that billions of people are using his work on a daily basis on their smartphone? CV: https://people.idsia.ch/~juergen/cv.html
> BTW, I committed a similar error in 1987 when I published what I thought was the first paper on Genetic Programming (GP), that is, on automatically evolving computer programs[GP1][GP] (authors in alphabetic order). At least our 1987 paper[GP1] seems to be the first on GP for codes with loops and codes of variable size, and the first on GP implemented in a Logic Programming language. Only later I found out that Nichael Cramer had published GP already in 1985[GP0] (and that Stephen F. Smith had proposed a related approach as part of a larger system[GPA] in 1980). Since then I have been trying to do the right thing and correctly attribute credit.
Source: https://people.idsia.ch/~juergen/deep-learning-miraculous-ye...
B: Priority disputes with Dr. Bengio (original date v Bengio's date): B1: Generative adversarial networks or GANs (1990 v 2014) B2: Vanishing gradient problem (1991 v 1994) B3: Metalearning (1987 v 1991) B4: Learning soft attention (1991-93 v 2014) for Transformers etc. B5: Gated recurrent units (2000 v 2014) B6: Auto-regressive neural nets for density estimation (1995 v 1999) B7: Time scale hierarchy in neural nets (1991 v 1995)
H: Priority disputes with Dr. Hinton (original date v Hinton's date): H1: Unsupervised/self-supervised pre-training for deep learning (1991 v 2006) H2: Distilling one neural net into another neural net (1991 v 2015) H3: Learning sequential attention with neural nets (1990 v 2010) H4: NNs program NNs: fast weight programmers (1991 v 2016) and linear Transformers H5: Speech recognition through deep learning (2007 v 2012) H6: Biologically plausible forward-only deep learning (1989, 1990, 2021 v 2022)
L: Priority disputes with Dr. LeCun (original date v LeCun's date): L1: Differentiable architectures / intrinsic motivation (1990 v 2022) L2: Multiple levels of abstraction and time scales (1990-91 v 2022) L3: Informative yet predictable representations (1997 v 2022) L4: Learning to act largely by observation (2015 v 2022)
LeCun claims four "main original contributions" and Schmidhuber basically debunks them one by one, for example:
> (IV) your predictive differentiable models "for hierarchical planning under uncertainty" - you write: "One question that is left unanswered is how the configurator can learn to decompose a complex task into a sequence of subgoals that can individually be accomplished by the agent. I shall leave this question open for future investigation."
> Far from a future investigation, I published exactly this over 3 decades ago: a controller NN gets extra command inputs of the form (start, goal). An evaluator NN learns to predict the expected costs of going from start to goal. A differentiable (R)NN-based subgoal generator also sees (start, goal), and uses (copies of) the evaluator NN to learn by gradient descent a sequence of cost-minimizing intermediate subgoals [HRL1].
It will be interesting to follow this.
When Fukushima published ReLUs in 1969 and CNNs in 1979, there were neither decent computers nor competitions. No excuse for not citing him.
> ReLU is an absurdly simple gate. The question revolved around its effectiveness, which was proven by Hintons lab.
Many good things are simple. They should have cited the creator, no matter how much they profited later from faster computers or novel datasets or the like.
> The key result is the outstanding performance on imagenet. If Schmidhuber was the actual pioneer, why wasn't he able produce the same results before Hinton?
Did his team ever participate in imagenet? Apparently not. He writes about DanNet: "For a while, it enjoyed a monopoly. From 2011 to 2012 it won every contest it entered, winning four of them in a row (15 May 2011, 6 Aug 2011, 1 Mar 2012, 10 Sep 2012)"
> NN were known to work for hand writing recognition since the 90s (including papers by Hinton). Dannet being able to do it for Chinese characters in 2010s is unremarkable.
The remarkable thing is that "DanNet was the first pure deep CNN to win computer vision contests." Before DanNet, other methods won the competitions. DanNet changed that.
However, the CNN pioneer was Fukushima who introduced the CNN architecture and ReLUs. Hinton did not cite him.
> You are well aware that not citing an earlier paper with different implementation and results is not plagiarism. There is absolutely no evidence of plagiarism anywhere.
So what exactly constitutes plagiarism? It's not about good or bad faith, it's about checking who did it first. If you are using building blocks from previous papers, you must cite them. Schmidhuber cites the difference between unintentional [PLAG1] and intentional plagiarism [FAKE2]:
[PLAG1] Oxford's guidance to types of plagiarism (2021). Quote: "Plagiarism may be intentional or reckless, or unintentional."
[FAKE2] L. Stenflo. Intelligent plagiarists are the most dangerous. Nature, vol. 427, p. 777 (Feb 2004). Quote: "What is worse, in my opinion, ..., are cases where scientists rewrite previous findings in different words, purposely hiding the sources of their ideas, and then during subsequent years forcefully claim that they have discovered new phenomena."
More quotes: "If one "re-invents" something that was already known, and only becomes aware of it later, one must at least clarify it later, and correctly give credit in follow-up papers and presentations." ... "And the authors did not cite the prior art - not even in later surveys."
This is crucial. Even later they did not cite the original sources.
> Following up on your logic is absurd, because I can conveniently state that back prop is just the chain rule in differentiation by Newton and everyone else has plagiarized from him.
The paper apparently both anticipated and corrected your claim (it wasn't Newton): "Some claim that "backpropagation is just the chain rule of Leibniz (1676) & L'Hopital (1696)." No, it is the efficient way of applying the chain rule to big networks with differentiable nodes (there are also many inefficient ways of doing this). It was not published until 1970.[BP1]"
[BP1] S. Linnainmaa. The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. Master's Thesis (in Finnish), Univ. Helsinki, 1970. See chapters 6-7 and FORTRAN code on pages 58-60. PDF. See also BIT 16, 146-160, 1976. Link. The first publication on "modern" backpropagation, also known as the reverse mode of automatic differentiation.
> And ReLU was plagiarized by Fukushima from neuroscience researchers.
Really? Can you prove this? Do you have a reference?
> 8. LBH devote an extra section to rectified linear units (ReLUs), citing papers of the 2000s by Hinton and his former students, without citing Fukushima who introduced ReLUs in 1969[RELU1-2] (see Sec. XIV).
This is only one of many concrete examples given.
DanNet obviously worked on all kinds of image data, otherwise it would not have won all those competitions before the similar AlexNet. However, the CNN pioneer was Fukushima who introduced CNNs and ReLUs.
Here is his take on what happened: https://people.idsia.ch/~juergen/scientific-integrity-turing...
What exactly did Post[POS] and Turing[TUR] do in 1936 that hadn't been done earlier by Gödel[GOD][GOD34] (1931-34) and Church[CHU] (1935)? There is a seemingly minor difference whose significance emerged only later. Many of Gödel's instruction sequences were series of multiplications of number-coded storage contents by integers. Gödel did not care that the computational complexity of such multiplications tends to increase with storage size. Similarly, Church also ignored the spatio-temporal complexity of the basic instructions in his algorithms. Turing and Post, however, adopted a traditional, reductionist, minimalist, binary view of computing—just like Konrad Zuse (1936).[ZU36] Their machine models permitted only very simple elementary instructions with constant complexity, like the early binary machine model of Leibniz (1679).[L79][LA14][HO66] Emil Post They did not exploit this back then—for example, in 1936, Turing used his (quite inefficient) model only to rephrase the results of Gödel and Church on the limits of computability. Later, however, the simplicity of these machines made them a convenient tool for theoretical studies of complexity.
An automatic theorem prover is a kind of AI. Gödel showed its limitations.
There have been entire conferences dedicated to Gödel and AI.