nicklecompte

There's a core problem with LLMs: they learn sentences, not facts.

So an LLM may learn a ton of English-language sentences about cats, and much fewer Spanish sentences about gatos. And it even learns that cat-gato is a correct translation. But it does not ever figure out that cats and gatos are the same thing. And in particular, a true English-language fact about cats is still true if you translate it into Spanish. So the LLM might be a genius if you ask it about cats in English, but in Spanish it might tell you "gatos tienen tres patas" simply because OpenAI didn't include enough Spanish biology books. These machines are just unfathomably dumb.

My very first response from gpt2-chatbot included a fictional source :(

> A study by Lucon-Xiccato et al. (2020) tested African clawed frogs (Xenopus laevis) and found that they could discriminate between two groups of objects differing in number (1 vs. 2, 2 vs. 3, and 3 vs. 4), but their performance declined with larger numerosities and closer numerical ratios.

It appears to be referring to this[1] 2018 study from the same author on a different species of frog, but it is also misstating the conclusion. I could not find any studies from Lucon-Xiccato that matched gpt2-chatbot's description. Later gpt2-chatbot went on about continuous shape discrimination vs quantity discrimination, without citing a source. Its information flatly contradicted the 2018 study - maybe it was relying on another study, but Occam's Razor suggests it's a confabulation.

Maybe I just ask chatbots weird questions. But I am already completely unimpressed.

[1] https://www.researchgate.net/profile/Tyrone-Lucon-Xiccato/pu...

Not in the sense of an architect actually building a generative AI blueprint. But transformer ANNs are simply incapable of designing physically plausible construction plans. Theoretically, not being able to solve graph connectivity seems like a big issue. Empirically, not being able to count is a fatal flaw.

You’re the one who said “starve to death” in your original comment! Maybe that’s a hint as to what the fear is: mass unemployment without universal basic income.

This is one of the most bizarrely bad-faith comments I’ve ever seen.

I think this is a clear case where “human-level” is a totally meaningless expression, since you have to ask about which humans[1]. On one hand, these were CS undergrads, so naively I would expect their score to be at least a bit higher than the white-collar worker average. On the other hand, these undergrads weren’t actually familiar with most of the software being tested - surely less familiar than GPT-4! And on the other-other hand I would expect an actual IT professional to do better. There is a difficult conflation with subject-matter knowledge and actual intelligence which makes “human-level” especially problematic. “CS undergrad-level” doesn’t roll off the tongue, but “human-level” seems downright misleading.

Nevertheless, this is a striking result, indicating categorical weakness in out-of-distribution reasoning. And I think these tasks being so familiar - real human problem-solving where your ability to think makes up for gaps in knowledge - helps illustrate the problem with transformers: they’re just not smart enough.

[1] I get very annoyed with discussions about “AGI” for systems that are totally incapable of solving problems that hunter-gatherers need to solve. But that’s another rant.

The claim is not really based on Tesla's technology - the problem is Tesla's dishonest advertising and Elon Musk being an unethical idiot.

Your rule makes some sense as a guideline, but it only works in the case where the driver and the auto manufacturer are doing the right thing yet something goes wrong. It does not work if the driver was watching YouTube instead of paying attention, and it does not work if the manufacturer routinely advertises a L2 system as having L4 capabilities. Even the names "Autopiliot" and "Full-Self Driving" are misleading. IMO its so misleading that Musk deserves to go to jail.

I couldn't get a clear sense either. But their website's focus on "orbit servicing" and EOL gives me the impression that this is really for proactively managing space debris from recent missions, not cleaning up historic debris. Their method makes more sense for cleaning dense "packets" of debris whose orbits haven't yet significantly deviated from the original mission orbit, so the cleanup satellite doesn't have to make too many maneuvers.

It still doesn't seem economical or even all that feasible.

"Bullshit" doesn't work because it requires a psychological "intent to persuade," but LLMs are not capable of having intentions. People intentionally bullshit because they want to accomplish specific goals and adopt a cynical attitude towards the truth; LLMs incidentally bullshit because they aren't capable telling the difference between true and false.

Specifically: bullshitters know they are bullshitting and hence they are intentionally deceptive. They might not know whether their words are false, but they know that their confidence is undeserved and that "the right thing to do" is to confess their ignorance. But LLMs aren't even aware of their own ignorance. To them, "bullshitting" and "telling the truth" are precisely the same thing: the result of shallow token prediction, by a computer which does not actually understand human language.

That's why I prefer "confabulate" to "bullshit" - confabulation occurs when something is wrong with the brain, but bullshitting occurs when someone with a perfectly functioning brain takes a moral shortcut.

It seems to me like the moon is "chatbots which are somewhat convincing" and everybody is landing there in OpenAI's wake. The real problem is Mars - make a computer which can learns as quickly and reason as deeply as, say, a stingray or another somewhat intelligent fish[1].

[1] This task seems far beyond the capability of any transformer ANN absent extensive task-specific training, and it cannot be reasonably explained by stingray instinct: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8971382/

This is pure speculation: I suspect our ancestors used music + simple words for communication long before we evolved modern human language. Although language outstripped music in terms of its precision and utility, as an evolutionary hangover we have a preference for language with musical qualities.

Some """evidence""" for this claim:

- The most interesting fact is that gibbons sing similarly to humans, intentionally targeting a pure tone so other gibbons can harmonize in octaves. They seem to use this for bonding and communicating their vocal state. So there's precedent for musical communication in one of our closest relatives.

- Music seems useful for hominids: just like modern humans, our pre-language ancestors possibly used "work songs" as a way for large groups of foragers to ping their status. Prairie dogs do the same with their chattering - if one critter goes quiet, the rest quickly realize something is wrong. There is no evidence that human language evolved after human music, but it seems more reasonable to me than the alternative.

- On the other hand, it's well-known that modern humans can be easily misled by speakers who use intellectual affectations to spritz up nonsense, and in general we tend to make lazy judgments about speakers and accept/deny their arguments accordingly. So, assuming this quirk didn't evolve very recently, I could see how "uses language musically" has the same (dishonest) social signal of intelligence as "uses big words all fancy-like," simply because 400,000 years ago the most musically talented hominids commanded the most social authority (even maybe when they shouldn't).

I said "confabulate" in my original post. "Confabulation" is a neurological symptom commonly seen in dementia patients, where a person isn't telling the truth because of errors in memory formation/recall or some other non-psychological problem in the brain. In particular, people who confabulate aren't aware their words are false and therefore it doesn't make sense to say that they're "lying." Likewise it's a problem with memory, not perception, so "hallucination" doesn't work either.

"Confabulation" still isn't great because humans confabulate with non-verbal memories and then express those confabulations in words; human confabulation mostly affects biographical memory, not subject matter knowledge. But considering how weird it is to even be talking about "memory" with a being that isn't aware of the passage of time, I think "confabulate" is the best option short of inventing a brand new word.

Oops! Good catch. I'm not an expert - I read "tangled type theory" once and "TTT" about 75 times....

That's not the point of the Gilligan's Island test. If the LLMs had said something to the effect of "I don't know, I wasn't trained to answer trivia about television" then I would agree with your comment.

Instead they all confabulated! Either they made up an episode title or insisted the episode doesn't exist. That's a serious problem: if the LLM doesn't properly recognize that it doesn't know something, how will it properly recognize it needs to use a tool to obtain that knowledge?

It seems like the only answer we have is for a user to flag that GPT-4+Web flubbed the Gilligan test, OpenAI dispatches a data contractor for RLHF whack-a-mole + synthetic data generation, GPT learns to Google answers to Gilligan's Island prompts, then we cross our fingers and hope transformers are smart enough to transfer that knowledge to the Sanford and Son benchmark.

I think the subtlety is that cephalopods/corvids/etc probably don’t hold false beliefs about the following:

a) things they care about

b) things they are cognitively capable of understanding to be false, and

c) things they can plausibly falsify for themselves by obtaining the relevant information.

So an octopus can have all sorts of misconceptions: indifferently making false assumptions about a cave they aren’t going to explore themselves; bizarre pseudo-magical explanations about human technology far beyond their understanding; incorrect guesses as to another octopus’s mental state. But what makes a conspiracy theory more than a mere misconception is the effort spent reinforcing the falsehood, using tools and methodologies which reflect a theoretical capability to understand why the conspiracy theory must be false[1]. (In particular the problem of people who cynically spread conspiracy theories without believing them.)

Conspiracy theories and other misconceptions motivated by politics or ideology are unique to humans because language + complex social organization + some intangible “humanness” means that abstract ideas and information can have great social power, and hence power in the real world, regardless of whether these ideas reflect reality.

[1] Note how many modern conspiracy theories are “backed up” by a slew of “evidence” which takes great analytical effort to refute line-by-line.

You absolutely could count the number of times "wizard" was used if you had the book in front of you. Similarly the LLM does have the chapter available "to look at." Documents pasted in the context window aren't ethereal.

This explanation/excuse doesn't hold water.

There is a major difference though: Holmes's proof was broadly comprehensible, just extremely complicated and easy to get lost in the details. In particular Holmes really tries to make the reader understand, with a fairly gentle/apologetic introduction.

But Mochizuki's "proof" is completely impenetrable even to experts on, like, page 3, and makes no effort to explain what is happening.

Another key difference is that New Foundations is a niche field, so there simply was not a huge amount of human effort spent reading Holmes's work. That's not the case with Mochizuki's proof. There's a big difference between "a small number of mathematicians didn't understand the proof but suspect it's correct" and "a large number of mathematicians didn't understand the proof and concluded it was incorrect."

And, most of all: Holmes's formulation of twisted type theory made the proof a natural candidate for dependently-typed formal verification. Mochizuki's proof is not type-theoretic and does not seem like a great candidate for the calculus of constructions - maybe it is! I actually have no idea. I suspect Mochizuki is the only person in the world who can answer that. But it's critical that Holmes did so much background work to simplify his proof and make this Lean program possible. Mochizuki should do the same.

AFAICT, both in terms of the "sociology of mathematics" and the amount of work required to even attempt a Lean formalization, trying to verify Mochizuki's proof is a waste of time.

One quibble I have with this: the items on the Joel Test are all language-independent, even if you're using a minor language with poor presence on GitHub. But that's not the case with AI codegen. GPT-3.5 was downright terrible at F#, and though Bing+GPT-4 does seem a lot better I'm not sure it's actually good enough for professional F#. I also doubt AI codegen is useful for Scheme/Racket developers who create DSLs (and of course it's totally useless for proprietary languages). So I am not sure AI codegen fits with the other items on the test.

In general I am concerned that LLMs will discourage innovation in programming language design - why write a better Python if GPT can just automate the tedium away?

A GPT-J chatbot talked a Belgian man into suicide last year: https://www.euronews.com/next/2023/03/31/man-ends-his-life-a...

And here's GPT-4/Copilot from this year: https://twitter.com/colin_fraser/status/1762351995296350592

I am extremely skeptical of LLMs and don't want to be mean to what's clearly a fun little experiment. But I thought that bit about "Claude 3 had the most fun playing the game" was strangely naive. Not just because these systems are incapable of emotions, but because Claude 3 was saying clearly "insincere" things like "wow, what a twist ending!" The reason why this is almost certainly "insincere" is that Claude 3 was trained on Wikipedia, including the Wikipedia article for this short, 24-year-old adventure game, which includes a full plot summary. So there are two possible things going on (not mutually exclusive)

1) Claude 3 is poor at knowledge transfer and can't connect the events of the game with a Wikipedia summary of those events.

2) Claude is simply imitating surprise because it noticed text that "looked like" a twist ending.

In general I wish the author had considered possible data contamination: how much does Claude 3 know about this game in a fresh session? If it can recite details from some online walkthrough then I am not sure this experiment is even meaningful.

Well, don't underestimate the sheer back-of-the-envelope power of a cat's brain: 700m neurons x 200Hz per neuron is a LOT of compute, like ~10,000 modern CPU cores in terms of ops/second. Pretty impressive when you consider the thermal efficiency....

Regardless, looking at AI power that way is a mistake. There are already a few supercomputers that can perform more operations/second than the human brain and I wouldn't be too surprised if this neuromorphic computer can perform more ops/s than a cat. But the structure of the brain is important: brains aren't just simply glued-together layers of neurons. The incredible topological complexity and integration of mammal brains surely means that certain "parallelizable algorithms" can be run much more efficiently, and that disparate information can be pulled together more readily for identifying patterns and making mental models. All mammals (and birds) are faster learners than any artificial neural network, and they understand ideas like object permanence and causality, which no AI is even close to getting right. It is reasonable to suppose that the structural complexity of mammal/bird brains is an important part why their cognitive abilities are so much greater than 2024 AI.

And I really can't overemphasize how complex mammal neurons are compared to artificial ones: https://www.quantamagazine.org/how-computationally-complex-i... It might not matter that individual biological neurons are slow when by themselves they're as sophisticated as deep artificial neural networks. A neuron might operate at 200Hz but it's not solving "what's the sum of these floating-point vectors?" like a CPU/GPU. It's solving something way more sophisticated: there was a paper recently arguing that humans had individual neurons for visually classifying 2 things, 3 things, etc. (Cats can do this "subitizing" almost as well as we can and probably have their own neurons for it, but their understanding of numbers seems shallower than a primate's.)

So:

- the artificial neurons are far simpler than cat neurons

- the links between the artificial neurons in neuromorphic computers are far simpler than the links between cat neurons in cat brains

- even in naive ops/second cats are not slouches

Simply matching the number of cat neurons and waving hands about "fast semiconductors" is not enough for me to think this machine is as powerful as a cat's brain. (And even if Intel made a computer theoretically capable of emulating a cat, I suspect it'll take several decades to develop AI software which is sophisticated enough to do the job.)

It's not even close to a mammal. It might have 1bn artificial neurons, while a cat has 700m biological neurons. But

a) individual mammal neurons are far more computationally complex than any existing artificial neuron, even in SNNs.

b) a cat has 10 trillion synapses compared to the 100 billion artificial synapses in Intel's computer. So even ignoring how simple artificial neurons are, mammal brains have far more topological complexity.

I have three basic problems with using "brain" and especially "human brain":

1) Nobody is even bothering to explain why this is like a human brain rather than 1M spider brains or 1B nematode brains - I suspect 1M spider brains is probably closer to reality, and I doubt that adds up to one human. The choice of "human" is a marketing gimmick.

2) Even in nematode brains, each neuron is very different, in a way that can't be explained by a small array of floating-point numbers. These SNN architectures are too homogenous and simple to explain the actual behavior of individual neurons. (See https://www.quantamagazine.org/how-computationally-complex-i... which discusses standard ANNs, but I strongly doubt a single layer of SNN neurons can actually emulate an individual rat neuron.)

3) Epigenetic effects are clearly important for thinking and memory formation, yet totally ignored by these architectures: https://www.smithsonianmag.com/smart-news/scientists-say-the...

My problem is that this behavior can be attained by a shallower and non-generalizable understanding. Instead of realizing that the hand is blocked, perhaps the system's model is equivalent to the hand disappearing and reappearing with a "swipe." This understanding would not be obtained by a 3D modeling of human anatomy, but rather a hyperfocused study of yoga videos where the camera angle is just like the one shown in the paper (it is mostly the cliched camera angle that raises my suspicions). An understanding like this would not always generalize properly, e.g. instead of a hand being partially occluded in a skew video it visibly pops in and out, or the left hand strangely blends into the right.

There's a general issue with generative AI drawing "a horse riding an astronaut" - art generators still struggle to do this because they just can't generalize to odd scenarios. I strongly suspect this method has a similar issue with "interpolate the frames of this yoga video with a moving handheld camera." AFAIK these systems are not capable of learning how 3D people move when they do yoga: they learn what 2D yoga instructional videos look like, and only incidentally pick up detailed (but non-generalizable) facts about 3D motion.

Great read - I learned quite a bit about this. The only quibble I had is at the end, and it's a very general layperson's view:

> But what’s even more impressive - extremely impressive - is that the system decided that the body would go up before going back down between these two poses! (Which is why it’s having trouble with the right arm in the first place! A feature matching system wouldn’t have this problem, because it wouldn’t realize that in the middle position, the body would go up, and the right arm would have to be somewhere. Struggling with things not visible in either input keyframe is a good problem to have - it’s evidence of knowing these things exist, which demonstrates quite the capabilities!).... This system clearly learned a lot about three-dimensional real-world movement behind the 2D images it’s asked to interpolate between.

I think that's an awfully strong conclusion to draw from this paper - the authors certainly don't make that claim. The "null hypothesis" should be that most generative video models have a ton of yoga instruction videos shot very similarly to the example shown, and here the AI is simply repeating similar frames from similar videos. Since this most likely wouldn't generalize to yoga videos shot at a skew angle, it's hard to conclude that the system learned anything about 3D real-world movement. Maybe it did! But the study authors didn't come to that conclusion, and since their technique is actually model-independent, they wouldn't be in a good position to check for data contamination / overfitting / etc. The authors seem to think the value of their work is that generative video AI is by default past->future but you can do future->past without changing the underlying model, and use that to smooth out issues in interpolation. I just don't think there's any rational basis for generalizing this to understanding 3D space itself.

This isn't a criticism of the paper - the work seems clever but the paper is not very detailed and they haven't released the code yet. And my complaint is only a minor editorial comment on an otherwise excellent writeup. But I am wondering if the author might have been bedazzled by a few impressive results.

> But today’s A.I. systems can pass the Turing Test with flying colors, and researchers have had to come up with new, harder evaluations.

It’s a bit petty to rag on a single sentence… but why do editors still let falsehoods like this slide? There is not a single LLM that can pass a properly administered Turing test. Just last week I saw GPT-4 badly failed a de facto Turing test because it wasn’t able to count to 17. The idea that a Turing test means “do laypeople find the dialogue eerily human-like?” is one of the tech community’s most pernicious bits of nonsense. And here it is, repeated uncritically in the New York Times. Extremely frustrating.

What this work actually shows is that a bunch of scientists are ignorant about how LLMs work, but are rushing to publish papers about them anyway. It is ridiculous for Thirunavukarasu to draw this conclusion from GPT-4's performance on a written ophthalmology exam.

From the good folks at AI Snake Oil[1]

> Memorization is a spectrum. Even if a language model hasn’t seen an exact problem on a training set, it has inevitably seen examples that are pretty close, simply because of the size of the training corpus. That means it can get away with a much shallower level of reasoning....In some real-world tasks, shallow reasoning may be sufficient, but not always. The world is constantly changing, so if a bot is asked to analyze the legal consequences of a new technology or a new judicial decision, it doesn’t have much to draw upon. In short, as Emily Bender points out, tests designed for humans lack construct validity when applied to bots.

> On top of this, professional exams, especially the bar exam, notoriously overemphasize subject-matter knowledge and underemphasize real-world skills, which are far harder to measure in a standardized, computer-administered way. In other words, not only do these exams emphasize the wrong thing, they overemphasize precisely the thing that language models are good at.

Also[2]:

> Undoubtedly, AI and LLMs will transform every facet of what we do, from research and writing to graphic design and medical diagnosis. However, its current success in passing standardized test after standardized test is an indictment of what and how we train our doctors, our lawyers, and our students in general. ChatGPT passed an examination that rewards memorizing the components of a system rather than analyzing how it works, how it fails, how it was created, how it is maintained. Its success demonstrates some of the shortcomings in how we train and evaluate medical students. Critical thinking requires appreciation that ground truths in medicine continually shift, and more importantly, an understanding how and why they shift. Perhaps the most important lesson from the success of LLMs in passing examinations such as the USMLE is that now is the time to rethink how we train and evaluate our students.

[1] https://www.aisnakeoil.com/p/gpt-4-and-professional-benchmar...

[2] https://journals.plos.org/digitalhealth/article?id=10.1371/j...

I don't have a good analytical response to this. Your comment doesn't pass the smell test for the following reason: I am a fairly serious jazz guitarist, and it seems to me like most great guitarists can be quickly identified by ear after playing three notes: Charlie Christian, Joe Pass, Blind Willie Johnson, Mary Halvorson, Django Reinhardt, etc. In some cases just one note is enough: B.B. King, Wes Montgomery, T-Bone Walker, Jimi Hendrix. (60 years later and still nobody plays octaves as well as Wes did).

Obviously some of their sound is tied to specific guitar models, but their tone is still easily identified on other guitars. And although these players were creative improvisers, that's not what I mean. Simply put: if it's a great guitarist, you can hear their hands like you can hear a singer's mouth. (Though hands are clumsy compared to mouths, and guitars are clumsy compared to horns.)

I am not anti-synth or anti-electronic music. But I do not believe any existing synth controller provides the expressive possibilities of an electric guitar, even Allan Holdsworth's goofy breath controller. Maybe I'm wrong and there's a great example out there on YouTube. But it seems to me that for an individual to express their individuality with a synth, they need to play a longer musical fragment and color their expression with timing, volume, vibrato, etc. Three notes "with pizazz" will not really be enough.

I understand your point but I think it's misplaced here. This kind of criticism made sense with the Bing/Sydney stuff, since "Bing claims to be self-aware" is misleading compared to "Bing inconsistently recites some sci-fi cliches." People like Kevin Roose were actively spreading misconceptions about how LLMs work. It would have been nice if they had taken your comment seriously in 2023.

But this is different. The subtext of the headline is clearly "Facebook's dumb chatbot had a very dumb glitch." I believe laypeople would immediately understand the AI is just plagiarizing a Facebook mom. Policing the language here seems more about pedantry than correcting actual misconceptions.

(You might say that some people would read this headline and jump to a Her fantasy, where Meta's poor AI is desperate for human connection or whatever. But these people are not going to be swayed by technical accuracy. They will just interpret language like yours as euphemism and denial.)

> The only control the piano gives you over note timbre is how hard you press the key.

This is just not true at all. The action of a piano is a very sophisticated mechanism that conveys both acceleration and velocity, along with the initial position of the key (e.g. fully released or half-pressed). This allows for a huge amount of tonal variation, though it's more subtle than a ribbon controller, and obviously a synth can do many things than a piano cannot.

"everything is possible" except for a synthesized instrument that actually has the expressive range of a real violin or saxophone. Even a synthesized piano is noticeably lifeless compared to the real thing (synthesized piano bass notes are especially thin).