Hacker News Reader

Comment by bonoboTP | Hacker News Reader

This sounds right to me as well but the point of this thread here is about how amazed one should be that this page was written in 2016. I'd say moderately, because maybe you woke up to AI being a thing this year or the last few, but the ideas obviously go back far and in surprisingly prescient ways. But this is not something unique to these AI labs. Obviously most laypeople will see the press releases and the media announcements as the big milestones and flagposts, but the academic research was inseparable from it. The AI labs obviously took on PhDs who trained in academia and had deep roots from those ideas. The story that some genius Sama invented this back when nobody thought about AI is the same exaggeration as Bill Gates inventing personal computing in his garage out of nowhere.

View on HN

Honestly, I was always one of the skeptics who thought AI was a probably-not-in-our-lifetimes moonshot (imagining that it couldn't be done on existing computer architectures) and that stuff like this article was just a bunch of puffery from people who read too much sci-fi.

But in the end, a lot of it was proven right, even if it wasn't quite in the way we imagined.. A lot of pre-LLM interpretations of AI imagine it as some sudden 0 to 100 breakthrough, like one day someone writes an AGI program in their basement and takes over the world with it. There's shades of that in here too, talking about worries of organizations secretly developing AI capabilities and needing to track public data for patterns to discover it. In the end there was a 'magic program' in the transformer, but it doesn't seem like they really foresaw how the program would be useless on its own, and the 'AI' would come from ingesting as much data as possible, a process that has built incrementally over years and been very much exposed to the public.

Exactly. What wasn't really foreseen is that it would come not through some big modeling insight but mainly through massive data scale. People much more imagined some clever general, compact learning algorithm, not quite just gradient descent, but something more intellectually satisfying, and something where you'd feel like "you cracked the mechanism" and you'd see clearly that some critical missing piece had to be invented that unlocked the "understanding" in the model, maybe some kind of fancy Hofstadterian self-referential loop or something. But it just turned out to be data and compute and of course engineering the algorithms to be efficient (which I don't want to discount of course).

That's basically the bitter lesson. Academics only reluctantly swallowed that pill and still aren't satisfied with this answer. It's ugly and feels like it shouldn't work because intuition would say there are too many combinations, curse of dimensionality, etc. But it turns out it's just line go up, extrapolate Moore's law and don't worry too much about philosophical-level breakthroughs just count the flops and bits. Ray Kurzweil's scifi extrapolations turned out closer to the truth, whether deservedly or by luck.

Reply on news.ycombinator.com