lsy

On the one hand, I actually thought it was pretty refreshing to see a tech company keynote that wasn't wall-to-wall with just-so futuristic AI demos that will, if they ever arrive, inevitably be unreliable and finicky under real world usage. Apple has a pretty big opportunity to be a company that invests in its core product functionality and to add AI strategically where it makes sense, keeping its products performant and stable while everyone else sloppifies.

But I don't think they are consciously doing that. I assume like every other company they are throwing a lot of resources at the new hype and just failing, and this is the backup plan. I'm not sure what that says about the strategy at the top.

If you’re really confident that AI is the future, what’s the use of belittling people who aren’t there yet? This kind of response feels more defensive than anything else.

Part of the reason the debate over usefulness is so exhausting is because there is an element of randomness, anecdote, and special pleading built into the use of the tool. One person has a good experience in their context, one person has a bad experience in their context. One person sees a bad result and throws up their hands and says it sucks, another tweaks their prompt until they get a good result and says it works.

In addition there's an element of personality: One person might think it's more meaningful to compose a party invitation to friends personally, another asks ChatGPT to write it to make sure they don't miss anything.

As someone who is on the skeptic side, I of course don't appreciate the "holding it wrong" discourse, which I think is rude and dismissive of people and their right to choose how they do things. But at the end of the day it's just a counterweight to the corresponding discourse on the skeptic side which is essentially "gross, you used AI". What's worse though is that institutions are forcing us to use this type of tool without consent, either through mandates or through integrations without opt-outs. To me it doesn't meet the bar for a revolutionary technology if it has to be propagated forcibly or with social scolding.

I think people hear "next token prediction" and think someone is saying the prediction is simple or linear, and then argue there is a possibility of "intelligence" because the prediction is complex and has some level of indirection or multiple-token-ahead planning baked into the next token.

But the thrust of the critique of next-token prediction or stochastic output is that there isn't "intelligence" because the output is based purely on syntactic relations between words, not on conceptualizing via a world model built through experience, and then using language as an abstraction to describe the world. To the computer there is nothing outside tokens and their interrelations, but for people language is just a tool with which to describe the world with which we expect "intelligences" to cope. Which is what this article is examining.

It’s honestly a little discouraging to me that the state of “research” here is to make up sci fi scenarios, get shocked that, e.g., feeding emails into a language model results in the emails coming back out, and then write about it with such a seemingly calculated abuse of anthropomorphic language that it completely confuses the basic issues at stake with these models. I understand that the media laps this stuff up so Anthropic probably encourages it internally (or seem to be, based on their recent publications) but don’t researchers want to be accurate and precise here?

One thought technology for understanding "hallucination" is that LLMs can only predict a fact statistically using all of the syntax available in its training data. This means that when you ask for a fact, you are really asking the computer to "postcast", or statistically predict the past, based on its training data.

That's why it "hallucinates", because sometimes the prediction of the past is wrong about the past. This differs from what people do, in that we don't see the past or present as a statistical field, we see them as concrete and discrete. And once we learn a sufficiently believable fact we generally assume it to be fully true, pending information to the contrary.

I feel like people in the comments are misunderstanding the findings in the article. It’s not that people save time with AI and then turn that time to novel tasks; it’s that perceived savings from using AI are nullified by new work which is created by the usage of AI: verification of outputs, prompt crafting, cheat detection, debugging, whatever.

This seems observationally true in the tech industry, where the world’s best programmers and technologists are tied up fiddling with transformers and datasets and evals so that the world’s worst programmers can slap together temperature converters and insecure twitter clones, and meanwhile the quality of the consumer software that people actually use is in a nosedive.

While the technology is indeed incredible, the question is not whether someone, somewhere, will find it useful for something, but whether the sorts of things it's useful for will economically justify the massive expenditure in both financial and human capital this trend is currently soaking up.

E.g. if "undergraduate-level reports" were something there was a mass market for, the economics of university education would be pretty different. And the same goes for idle searches, sycophantic therapizing, blog article generation, and toy code development: there is a solid user base while costs are free or low, but that says little about whether there is an appetite to pay for these tools, especially if the prices are commensurate with the cost of operation.

What's astonishing to me is the number of companies that will supposedly hire someone and give them credentials without even seeing them on camera. The proliferation of this narrative seems somewhat real and somewhat calculated to further undermine the legitimacy of remote work. But you would think something like "in-person orientation" and requiring that people use their cameras in meetings would solve a lot of the issues here.

The point is well-made that truly revolutionary technologies don't need top-down mandates to drive adoption, and it's certainly a bad sign for the AI industry that we are in this phase of the hype cycle. But it also seems likely that top-down mandates will create perverse incentives for product development around these technologies, preventing genuinely valuable use cases from emerging as people determine what does and doesn't work.

If everyone, to satisfy their CEO's emotional attachment to AI, is forced to type into a chat box to get dreck out and then massage it into something usable for their work, we'll see that ineffective mode persist longer, and probably miss out on better modes of interaction and more well-targeted use cases.

This is unnecessarily self-congratulating. The problem is that vulnerabilities are found in cars after they are on the market for a while and already purchased, so existing owners get their rates hiked, but the manufacturer never fixes the issue. No amount of research is going to guarantee your operating cost next year.

I don’t see a palpable effect from “AI”, I think software as an industry is just extremely sensitive to interest rates.

When money is cheap every company in every industry can soak up engineers to make speculative or vanity apps and websites. When money gets tight companies find ways to do without. So on top of software-qua-software companies decreasing hiring, other companies also hire fewer engineers, and the effect compounds.

The article puts scare quotes around "understand" etc. to try to head off critiques around the lack of precision or scientific language, but I think this is a really good example of where casual use of these terms can get pretty misleading.

Because code LLMs have been trained on the syntactic form of the program and not its execution, it's not correct — even if the correlation between variable annotations and requested completions was perfect (which it's not) — to say that the model "understands nullability", because nullability means that under execution the variable in question can become null, which is not a state that it's possible for a model trained only on a million programs' syntax to "understand". You could get the same result if e.g. "Optional" means that the variable becomes poisonous and checking "> 0" is eating it, and "!= None" is an antidote. Human programmers can understand nullability because they've hopefully run programs and understand the semantics of making something null.

The paper could use precise, scientific language (e.g. "the presence of nullable annotation tokens correlates to activation of vectors corresponding to, and emission of, null-check tokens with high precision and accuracy") which would help us understand what we can rely on the LLM to do and what we can't. But it seems like there is some subconscious incentive to muddy how people see these models in the hopes that we start ascribing things to them that they aren't capable of.

The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM bespeaks the lack of rigor in this field. The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it! They aren't references to internal concepts, the model is not aware that it's doing anything so how could it "explain itself"?

CoT improves results, sure. And part of that is probably because you are telling the LLM to add more things to the context window, which increases the potential of resolving some syllogism in the training data: One inference cycle tells you that "man" has something to do with "mortal" and "Socrates" has something to do with "man", but two cycles will spit those both into the context window and lets you get statistically closer to "Socrates" having something to do with "mortal". But given that the training/RLHF for CoT revolves around generating long chains of human-readable "steps", it can't really be explanatory for a process which is essentially statistical.

Assuming this result holds, and knowing that LLMs (including 4o) nevertheless remain incapable of standing in for people in most cases that require intelligence, this seems like a damning indictment of the test as an indicator of genuine intelligence.

In addition to Kamshak's note about parallel inference accumulating float errors differently due to order of operations, which makes LLMs theoretically non-deterministic at temperature 0, there is the issue of them being practically non-deterministic as-deployed, not just via temperature but because of inclusion of prior "turns" in context, variations in phrasing of prompts, etc.

It's also "non-deterministic" in the sense that if you removed all sources of non-determinism and asked "What is 1+1?" and received the answer "2" deterministically, that doesn't guarantee a correct answer for "What is 1+2?". Ie a variation in the input isn't correlated in a logical way with a variation in the output, which is somewhat fatal for computer programs, where the goal is to generalize a problem across a range of inputs.

One effect of this technology being almost totally opaque, non-explainable, and also nondeterministic is that most of the arguments for it (and maybe most against it?) take on an anecdotal character.

I was surprised to hear Dr. Bubeck, who is in kind of a privileged position wrt OpenAI and obviously an accomplished scientist, essentially saying things like "I tried asking it X and I am pretty sure it's not in the training set, and it worked, therefore I think it understands".

A really big problem with the anecdata approach to proving out AI is alluded to in Dr. Bender's story about the "Everything in the Whole Wide World" museum (for those who didn't watch, Grover the muppet goes to the aforementioned museum, sees many things — but not "everything", then walks through a door labeled "Everything Else" that leads outside). That is, no individual prompt-and-response can be relied on to inform about another prompt-and-response that you haven't yet tried. And with nondeterminism, you can't even rely on that prompt-and-response to remain stable. But a failed prompt also carries the possibility that a tweaked prompt could produce a success. So (as we saw with the twitter post yesterday complaining about LLMs not working well that was both heavily upvoted and heavily contested in the comments), we are in a world where nobody can say much except to provide anecdotes of the thing failing or succeeding or having characteristic Y, which are exercises in narrative construction to support or oppose investment, not principled discussion.

Thanks for commenting, I like the example because it's simple enough to discuss. Isn't it more accurate to say not that Claude "realizes it's going to say astronomer" or "knows that it's going to say something that starts with a vowel" and more that the next token (or more pedantically, vector which gets reduced down to a token) is generated based on activations that correlate to the "astronomer" token, which is correlated to the "an" token, causing that to also be a more likely output?

I kind of see why it's easy to describe it colloquially as "planning" but it isn't really going ahead and then backtracking, it's almost indistinguishable from the computation that happens when the prompt is "What is the indefinite article to describe 'astronomer'?", i.e. the activation "astronomer" is already baked in by the prompt "someone who studies the stars", albeit at one level of indirection.

The distinction feels important to me because I think for most readers (based on other comments) the concept of "planning" seems to imply the discovery of some capacity for higher-order logical reasoning which is maybe overstating what happens here.

The natural-language chat box (or voice command) is seductive as an interface because it elides the question of how somebody will use a product. It makes a product that uses it seem more potentially capable, because you can imagine asking it anything (we all remember the sense of possibility in the first Zork-style games). If the user was provided with an affordance to understand the range of capabilities of the product, they might realize it's too limited for them, or realize that their use case is served by something else.

The chat/voice interface is like Skinner's Box: we keep "using" because we don't know what will work and what won't, and when something works it seems magical. And it keeps being pushed on customers because it's beneficial for corporations: customers will fruitlessly navigate chat support rather than realizing they should give up and call, businesses will invest in products with open-ended interfaces because they can convince themselves it's flexible enough for their needs.

Going back to the original "Bitter Lesson" article, I think the analogy to chess computers could be instructive here. A lot of institutional resources were spent trying to achieve "superhuman" chess performance, it was achieved, and today almost the entire TAM for computer chess is covered by good-enough Stockfish, while most of the money tied up in chess is in matching human players with each other across the world, and playing against computers is sort of what you do when you're learning, or don't have an internet connection, or you're embarrassed about your skill and don't want to get trash-talked by an Estonian teenager.

The "Second Bitter Lesson" of AI might be that "just because massive amounts of compute make something possible doesn't mean that there will be a commensurately massive market to justify that compute".

"Bitter Lesson" I think also underplays the amount of energy and structure and design that has to go into compute-intensive systems to make them succeed: Deep Blue and current engines like Stockfish take advantage of tablebases of opening and closing positions that are more like GOFAI than deep tree search. And the current crop of LLMs are not only taking advantage of expanded compute, but of the hard-won ability of companies in the 21st century to not only build and resource massive server farms, but mobilize armies of contractors in low-COL areas to hand-train models into usefulness.

For CS-related books, I think the original 1979 Hopcroft and Ullman is better than its successor editions.

I don't know, the expiration of low interest rates seems to fully explain the slump in the software industry, without considering the advent of chatbot-assisted coding. The job market is just much smaller and new entrants will unfortunately have a tough time. We may be back to the days of de rigeur unpaid software internships, or college grads taking other, non-programming jobs.

But it seems like a bad sign for a technology when think pieces need to be written portraying skeptics as cro-magnons on the evolutionary scale of adoption, there's something very "emperor's new clothes" about it. The value of revolutionary tech typically speaks for itself and doesn't need extensive apologetics.

I'm also put off by the author's condescension towards people who aren't convinced after using the technology. It's not the user's job to find a product useful, it's a product's job to be useful for the user. If a programmer puts a high value on being able to trust a program's output to be minimally conformant to libraries and syntax that are literally available to the program, and places a high value on not having to babysit every line of code that you review and write, that's the programmer's prerogative in their profession, not some kind of moral failing.

Adopting this perspective would improve the quality of efforts around this technology. Instead of thinking of it as somehow creating an "intelligence", seeing it as a complex lens on the training data that is controlled by the prompt helps you understand that the output isn't generated by the model, but by people. And various existing pieces of human effort are brought into focus and collimated by nudging the lens in different directions with a "prompt". The user then gives those pieces meaning and determines whether the result is useful or not.

This makes certain things more clear: notions of "truth" are not in play beyond statistical happenstance, certain efforts to make outputs uniform are more trouble than they're worth, and valuable use cases are strongly correlated with the ability and convenience of the user to confirm the usefulness of the result.

I don't know that this is super well-founded: It seems similar to the "Fremen Mirage" [1], and misses that in most cases the society that escapes war for longer will have time and energy to build infrastructure and accumulate resources that provide a decisive advantage in conflict and defense. Looking back at history it's rare that the "virile fighting" nation/group wins against a more "civilized" adversary that's better organized and resourced.

[1] https://acoup.blog/2020/01/17/collections-the-fremen-mirage-...

What a bizarre fixation for a national paper, the follow-on letter from Will Lewis is even more transparently fixated on the daily appearance of the "pillars" [1]. "Personal liberties" appears to be a new construction that is conspicuously different from "civil liberties", and I think the distinction really speaks to what Bezos wants here. At the end of the day I expect the quality of discourse to dip: being afraid of contradictory opinions is the first step in intellectual decline.

[1] https://www.hollywoodreporter.com/business/business-news/jef...

Research skills involve not just combining multiple pieces of data, but also being able to apply very subtle skills to determine whether a source is trustworthy, to cross-check numbers where their accuracy is important (and to determine when it's "important"), and to engage in some back and forth to determine which data actually applies to the research question being asked. In this sense, "deep research" is a misleading term, since the output is really more akin to a probabilistic "search" over the training data where the result may or may not be accurate and requires you to spot-check every fact. It is probably useful for surfacing new sources or making syntactic conjectures about how two pieces of data may fit together, but checking all of those sources for existence, let alone validity, still needs to be done by a person, and the output, as it stands in its polished form today, doesn't compel users to take sufficient responsibility for its factuality.

It's hinted at in the article, but so far, it seems that 95%+ of use cases are satisfied by free-tier models, and will imminently be satisfied by local, open-weight models. It's not clear that people or businesses will ever be willing to pony up for the extra marginal accuracy when they can get most of the benefits for cheap or for free.

I'd actually say that in contrast to debates over informal "reasoning", it's trivially true that a system which only produces outputs as logits—i.e. as probabilities—cannot engage in *logical* reasoning, which is defined as a system where outputs are discrete and guaranteed to be possible or impossible.

I think part of this is that you can't trust the "thinking" output of the LLM to accurately convey what is going on internally to the LLM. The "thought" stream is just more statistically derived tokens based on the corpus. If you take the question "Is A a member of the set {A, B}?", the LLM doesn't internally develop a discrete representation of "A" as an object that belongs to a two-object set and then come to a distinct and absolute answer. The generated token "yes" is just the statistically most-likely next token that comes after those tokens in its corpus. And logical reasoning is definitionally not a process of "gaining confidence", which is all an LLM can really do so far.