Almost all statements Yann LeCun made back then (LLMs won't be able to do this or that) turned out to be wrong. They were right to fire him.
RCA developed the first good color TV cameras. They were monsters. Originally, electronic TV (as opposed to mechanical scanning) used Zworykin's iconoscope. This needed huge amounts of light. It didn't have a photo-multiplier stage to boost sensitivity, but it did accumulate charge from light over the whole frame time. There was an alternative approach, Farnsworth's image dissector, which also needed huge amounts of light. It had a photo-multiplier stage but only the light from the dot currently being scanned by the scanning beam mattered.
So RCA threw money at the problem. They developed the image orthicon camera tube. It combined the advantages of both approaches, integration over a full frame and a photo-multiplier stage. It was insanely expensive, costing thousands of dollars in the 1950s. It needed about twenty different voltages and multiple independent power supplies. It worked reasonably well.
Then came color. That took three image orthicons (RGB) inside a box the size of a small refrigerator, along with optics to split the incoming image into three colors. Plus a rack of analog electronics per camera, connected through a large cable. Plus an alignment problem from hell, with a huge number of adjustment knobs.[1]
Scaling worked! From the 1950s to the 1970s, color TV broadcasting was powered by those monsters.
Now, independent of this, other people were working on other approaches. Vidicons, silicon detectors inside camera tubes, and finally fully solid state image sensors. Eventually, in 1983, after thirty years of image orthicon monsters, RCA produced a usable solid state camera. Today, a cell phone camera costs about US$10 in bulk, and it's far better than the monsters.
It took thirty years of low-priority development to get past the giant image orthicon cameras.
Electronic fluorescent lamp ballasts had a different problem. That was a suppressed invention. The magnetic ballast industry was dominated by Magnetek. Someone invented a solid state replacement, which was smaller and cheaper, and licensed their patent to Magnetek for royalties. Magnetek just sat on it, not wanting to gut their core market. The inventor hired Townsend, Townsend and Crew in Palo Alto and sued. (Read this in the law firm's brag book in their office.) Eventually, the inventor got a hundred million or so, solid state ballasts went into production, and Magnetek seems to be gone. (The company called Magnetek today, which makes crane controls, is an offshoot of a Litton unit, and seems to be unrelated.)
There are also parallels from the steam locomotive industry, the Baldwin Locomotive Works being the best known. Baldwin actually built the first Diesel locomotive. They made a few half-hearted tries at Diesels over several decades, then gave up and went back to steam. Baldwin ceased all locomotive production in 1956. The corporation lingered on for a while, not doing much.
[1] https://www.tvcameramuseum.org/rca/tk40a/pix/rcatk40aart.pdf
Suppressed inventions are very rare, but that's one of them.
One of the few others is Airadar, from 1973.[1] Not "AI", "air radar". This was a phased-array radar for small aircraft. Way ahead of its time. So far ahead that the USAF was upset about it being available commercial before it was in use by the military.
[1] https://books.google.com/books?id=NWzlTqj0gQ4C&pg=PA64&lr=&r...
But even in computing, when there is a promising way forward that the basic reaearch has been done on, its hard to justify side quests.
How does that work when the research team moved elsewhere? Re-hire talent? That takes time - the 1 thing you don't have in a fast-moving field like AI. Post sums it up nicely:
"Do the math—how much is it worth to halve training and inference cost given that the industry has investments approaching a trillion dollars?"
How much is these researchers' salaries, compared to what MSFT spends on compute? Chances are it's peanuts in comparison.
Companies will try to ruthlessly rid themselves of labor as long as those members of labor have agency and rights.
If someone disagrees they can convince me by explaining why a company wouldn’t choose to spend X on a machine they wholly own vs X on an employee they don’t.
There’s some room for argument that employees can do more than machines can but in the current AI zeitgeist arguing that we have machines with equivalent to human intelligence I don’t see how you wouldn’t see this coming if you believed the zeitgeist.
...And because they have monopoly money. Basically, research labs are usually vanity spending for corporations with too much money. Xerox parc was funded by a monopoly. Bell labs was funded by a monopoly. Google funds its moonshots with its monopoly.
Ah, well, good thing we've almost entirely defunded academic science.
Seems weird when it's become apparent that LLMs aren't really special sauce outside of just having the compute.
He was against LLMs, reinforcement learning, etc. Claiming that those are a dead-end. He was wrong. His own "JEPA" ideas have basically gone nowhere. Meta was right in moving on without him.
10% chance to be market leader doesn’t sound as good as 60% second place to them for example
Once you get close to the top, the risks don't look worth it.
It's his call for sure, but rather funny as an argument.
It is because even if it would pan out, it would mean temporarily losing AI leadership, reducing stock price, losing the ability to hire talent, and by virtue of that losing the opportunity permanently.
Zuckerberg wanted to placate his shareholders with a shiny new thing now.
would've stopped Meta from delivering a polished product like Muse sooner.
Muse is not polished, unless usability and programming standards have fallen so far in the LLM vibe coding era that a pre-alpha state is now considered polished.
Muse is imperfect, as is all software, but calling it pre-alpha is a bit much.
AGI will require architecture changes not simply more data.
LLMs work well for search problems with the help of a strong verifier.
As it turns out, every problem is ultimately a search problem. But not every domain has an easily available verifier.
The author states "I consider the scaling-only focus to be both misdirected and not fun.". I get the point but also totally get the focus on scale out for now.
For a corporation it is to make money.
For a researcher it is to explore and illuminate. That this exploration and illumination often leads to a landfall of new wealth generation is a happy accident. This ensures that the monied class would bet some of their capital on them, at least the wise ones.
Where can I read more about this? I have seen people make this claim for a while, and when I looked deep, it used to turn out to be wrong.
Yann LeCun has the syndrome where someone who sees the value of B (non-text modalities, i.e. physical reality) cannot stop themselves from constantly critiquing A (LLMs), even though clearly A has its own advantages.
I think he is a smart guy. But he constantly deploys the crackpot's PR playbook.
Which is superior, the big f*king monstrosity or the elegant on-demand, small batches, reduce waste?
Meta will go in the shitter and Deepseek will make the small batches.
It does (Lean at least) incorporate controlled experimentation, but tightly controlled with a both a laser focus on what the goal is and tight resource constraints. And Lean (at least several writers on Lean) favors taking multiple paths with a a quick and low risk and a longer-payoff, higher-risk, higher reward path for some efforts—one source used the US Navy’s two-path approach to SLBMs with Polaris and Poseidon as a model for this—but that's different than “maybe there is something down this path” basic research, which is outside of the concern a production model like TPS/Lean and often completely outside the model of a product-oriented firm; basic research is something firms consume and incorporate more than generate, because until the basics have been proven out, the rationally perceived chance of any particular avenue paying off is near zero for any firm.
Toyota is also a good example of what not to do. Their lack of earlier research investment in EVs has led to being severely behind the market, in addition to their lack of focus on quality causing poorer sales (their quality used to be one of the only reasons to buy them). KIA basically stole their playbook by creating loads of good-quality cheap hybrids and EVs. Toyota could've had that market if the invested more ahead of time in research.
And that's what's happening with AI. While there are plenty of players now, it's the ones that aren't looking ahead that will be left behind. Nothing scales forever; when you hit the ceiling, you're trapped in a box of your own making.
The rest is history.
That failed and SpaceX was then forced to develop its own brand new engines.