6,586 karma · joined April 20, 2009
Are you proposing that the attention input context is gone, or that the attention mechanism’s context cost is computationally negated in some way, simply because the system processes natural language? Having the attention mechanism selectively isolate context on command would be an important technical discovery.
But the cloud is different. None of the financial scale benefits are passed on to you. You save serious money running it in-house. The arguments around scale have no validity for the vast, vast majority of use cases.
Vercel isn't selling bread: they're selling a fancy steak dinner, and yes, you can make steak at home for much less, and if you eat fancy steak dinners at fancy restaurants every night you're going to go broke.
So the key is to understand whether your vendors are selling you bread, or a fancy steak dinner, and to not make the mistake of getting the two confused.
This, combined with a subsequent reranker, basically eliminated any of our issues on search.
Coding is an interesting example because as we change levels of abstraction from the syntax of a specific function to, say, the architecture of a software system, the ability to measure verifiable correctness declines. As a result, RL-tuned LLMs are better at creating syntactically correct functions but struggle as the abstraction layer increases.
In other fields, it is very difficult to verify correctness. What is good art? Here, LLMs and their ilk can still produce good output, but it becomes hard to produce "superhuman" output, because in nonverifiable domains their capability is dependent on mimicry; it is RL that gives the AI the ability to perform at superhuman levels. With RL, rather than merely fitting its parameters to a set of extant data it can follow the scent of a ground truth signal of excellence. No scent, no outperformance.
He is right that non-RL'd LLMs are just mimicry, but the field already moved beyond that.
High minimum wages are making it nearly impossible to run small restaurants. We're seeing restaurant apocalypse happening in places like Seattle and Denver, because high wages result in high prices which causes lower customer volume which causes higher fixed costs per unit, which causes a death spiral. Denver, for example, has 30% fewer restaurants now, because it's so hard to run one profitably.
Lots of little tasty restaurants and high wages for service staff are basically incompatible. Many people in the city don't realize this and advocate for both.
Of course, high wages are desirable if they can be accomplished without tradeoffs, but the tradeoffs are there.
High minimum wages favor high-volume, fast service chain restaurants that are more labor efficient.
Perhaps eventually automation will relax this tradeoff, but I would expect automation to primarily benefit corporate restaurant chains over small local eateries, unless the automation is so general that any restaurant can start using it without technical expertise or R&D.
It's been some time since LLMs were purely stochastic average-token predictors; their later RL fine tuning stages make them quite goal-directed, and this is what has given some big leaps in verifiable domains like math and programming. It doesn't work that well with nonverifiable domains, though, since verifiability is what gives us the reward function.
I'm sympathetic to the idea that God started the whole shebang (that is, the universe), because it's rather difficult to disprove, but looking at the biological weight of evidence that brain structures evolved over many different species and arguing that something magical happened with homo sapiens specifically is not an easy argument to make for someone with any faith in reason.
The human brain is not that different. Our long-term memories are stored separately from our executive function (prefrontal cortex), and specialist brain functions such as the hippocampus serve to route, store, and retrieve those long term memories to support executive function. Much of the PFC can only retain working memory briefly without intermediate memory systems to support it.
If you squint a bit, the structure starts looking like it has some similarities to what's being engineered now in LLM systems.
Focusing on whether the model's weights change is myopic. The question is: does the system learn and adapt? And ICL is showing us that it can; these are not the stateless systems of two years ago, nor is it the simplistic approach of "feeding old context back to it."
Economics is the study of scarcity. Scarcity comes in many forms: land, attention, materials, time. (It is not the study of money, which is a common belief of those who have not thought about what money is. Money is merely a fungible token claim on resources, not the resource itself.)
You could also believe that the study of physics is just the study of business, because businessmen operate in a world governed by physics.
The creators and defenders of the second amendment were and are smart people. They were skeptical of highly concentrated power, which is why we have a federalist system; they were also highly skeptical of large concentrations of militarized police power, which is why our police in general are run by states and municipalities. It is really quite difficult to reconcile support of a massive ICE apparatus with any sort of comprehension of the political philosophy of the US constitution.
Is that true, or is there something more here?
Students don't get to choose whether to take the test, so why do we give AI the choice?
The trend has been resistant to any particular link to localized economic ups or downs. Characterizing the 2023-2025 era (at least in the US) as "a time of scarcity" is divorced from any sort of factual reality; there is no quantitative data to support this idea and it seems to mostly be based on social media vibes (hence the oft-commented "vibecession").
One could make a much stronger argument exactly to the opposite: wealthier societies tend to become more individualistic and separated, people choose to live on their own if possible, and in bigger places; large companies have such attractive economics and pay people so much more than small companies do that it is difficult for small organizations to compete for talent.
But then they find something unregulated and manage to flub it so badly it makes a strong case for regulation.
I'm not sure if Elon was involved in these door handles on the interior, though I do recall he was very involved in the early touch-to-expose exterior handles and insisted they use them despite internal engineering pushback. I think a lot of the bad engineering designs seem to originate from internal engineers getting overruled by Elon.
- I'm reviewing the last meeting of a regular meeting cadence to see what we need to discuss.
- I put it in a lookup (vector store, whatever) so I can do things like "what was the thing customer xyz said they needed to integrate against".
Those are pretty useful. But I don't usually read the whole meeting notes.
I think this is probably more broadly true too. AI can generate far more text than we can process, and text treatises on what an AI was prompted to say is pretty useless. But generating text not with the purpose of presenting it to the user but as a cold store of information that can be paired with good retrieval can be pretty useful.
To whatever extent you were to succeed, the rest of us would be worse for it. We need more Simons.
That's needed because many APIs are either nonexistent or extremely marginal in design and content coverage.
I know someone creating a game, and she is using AI in asset creation. But she's also a highly skilled environment technical artist from the industry, and so the use of AI looks totally different than all these demos from non-artists: different types of narrow AI get injected into different little parts of the asset creation pipeline, and the result is a mix of traditional tools (Substance Designer + Painter, Blender, Maya) with AI support in moodboarding, geometry creation and some parts of texture creation. The result is a 2-5x speedup, but instead of looking like slop it looks like a stylistically distinctive, cohesive world with consistent art direction.
The common pattern is that people think AI will automate "other people," because they see its shortcomings in their own field. But because they don't understand the technical skill required in other fields, they assume AI will just "do it." Instead, it seems like AI can be a force multiplier for technically skilled people, but that it begins showing its weakness when asked to take over entire pipelines normally produced by technically skilled people, whether they be engineers or artists.
But what you're arguing is different: that increasing supply has no effect at all on the clearing price. That would require an unusual demand "curve" that is perfectly flat, i.e., perfectly elastic, where there is infinite demand at a given price and zero demand at just a dollar above that price (or else that infinite demand would have already pushed prices up higher than the pre-existing price).
This clearly doesn't make any sense for the housing market; home buyers are sensitive to price, there is not infinite demand, some people have more or less desire to pay for a house. In fact, perfectly elastic markets essentially don't exist, and very low slope demand curves only exist in some unusual edge cases in markets (such as commodities that are near-perfect substitutes).
"Squeezing out" is done by a price mechanism: a family that would prefer to stay in Seattle decides to sell, because that new buyer (unable to buy the condo, because it hasn't been built) decides to offer a high enough price to induce the existing family to leave.
That's only done by reducing housing affordability (increasing prices) which is the public policy outcome we're trying to avoid.
It sounds like you agree that new supply is good, I think, because you believe new entrants would otherwise "squeeze out" existing residents and I assume you would agree that this is done by price, and so therefore you would also agree that new housing stock (which decreases the "squeezing") also suppresses price level relative to the alternative fixed stock scenario.