nicklecompte

It's not even "human knowledge" that can't be written down - it seems all vertebrates understand causality, quantity (in the sense of intuitively understanding what numbers are), and object permanence. Good luck writing those concepts down in a way that GPT can use!

In general AI in 2024 is not even close to understanding these ideas, nor does any AI developer have a clue how to build an AI with this understanding. The best we can do is imitating object permanence for a small subset of perceptible objects, a limitation not found in dogs or spiders.

His point is that "evolution by selection" also includes that transformers are easy to implement with modern linear algebra libraries and cheap to scale on current silicon, both of which are engineering details with no direct relationship to their innate efficacy at learning (though indirectly it means you scale up the training data for more inefficient learning).

This is irrelevant to the study, because the musical ear is attuned to human voices and that’s what instruments are trying to emulate. A specific tuning system is a sacrifice that human voices don’t have to make (e.g. the barbershop 7th versus a dominant 7th on a piano, or wolf intervals in Pythagorean tuning).

Pythagorean tuning makes a ton of sense if you’re trying to tune a fretted stringed instrument (and haven’t tried playing anything with a key change), but it’s an oversimplification of how music works. The Pythagorean philosophical mistake is treating the tuning system as a theory of music itself, when it’s really a simple framework for certain types of music.

I am glad studies like this exist but "we subjected human data analysts to 100 trite data analysis problems with unambiguous solutions, without any surrounding context or personal/professional incentive" is going to overestimate the relative efficacy of LLMs. Certainly one advantage they have over humans is that they don't get bored with SQL 101 questions.

Nothing about what you’re saying actually deals with this non-obvious limitation of chain-of-thought:

> Examples like this suggest that transformers wouldn’t gain much from using just a few intermediate steps. Indeed, Merrill and Sabharwal proved that chain of thought only really begins to help when the number of intermediate steps grows in proportion to the size of the input, and many problems require the number of intermediate steps to grow much larger still.

This aligns with my experience: GPT-4 can only break down “simple” problems when prompted to solve step-by-step. In particular, if the actual steps need to be broken down further (O(n^2) complexity), GPT-4 can’t handle it reliably - it will break a tasks into steps but it struggles to break subtasks into substeps even if it otherwise can solve the subtask with CoT prompting.

CoT prompting works for simple O(n) computations because it prevents LLMs from blindly guessing the answer, but they are theoretically (and IMO empirically) incapable of breaking any O(n^2) problem down into O(n) separate O(n) subproblems. Needless to say humans are quite a bit smarter than that. (so are mice!)

Yeah, too deep to keep watch over I think, unless they went for air in shifts. There's also the issue that if you're on land, gravity means you don't need an airtight container to cook/smelt/etc. But underwater it would be hard to prevent things from leaking to the sea.

I think mermaids could do fish kebabs over a gas vent (though they'd lose a lot of food), but they couldn't use it as a kiln.

Obviously you're not wrong about fire being a major blocking point for sea creatures[1] - hard to make metals without fire. But possibly mermaids could "cook" their food via fermentation, e.g. putting food in a sealed swim bladder, then burying it in the sand. This could fulfill the same brain calorie requirements.

[1] Of course, maybe in Andromeda some sea-hackers are reading Hacker Sea-News and arguing land-based animals could never develop civilization because something-something.

But if Smart Paste makes errors even 1% of the time then that would be astonishing from an LLM benchmark POV and still completely unacceptable from a CRM reliability POV. Nobody wants a data entry system where you have to fix 1 in 100 rows because the computer made an error.

Orcas are smart enough to give each other distinct names and seem to understand the concept of verbs/nouns when identifying human words[1]. They could very well be smarter than us, but they'd never dominate the planet technologically: their flippers have no evolutionary path to developing fingers and thumbs.

[1] Some people point out that we've never learned orca language but they've learned ours. This is true, but it's also true that orca scientists have never kidnapped human children and intentionally tried to teach them orca language. So it's more of a poetic comparison than a scientific one.

They’re good at summarizing text convincingly if you didn’t read the original document, but GPT-4 still makes a ton of basic factual mistakes: https://arxiv.org/abs/2402.13249

I think it will be several decades - maybe longer - before you can throw an arbitrary 20-page document into a computer program and get an accurate one-page summary. People need to take seriously that transformer LLMs don’t understand human language.

But we don't have a clue what "understanding" truly means when it comes to animals, including humans, which is the more relevant problem for this particular thread[1]. There is an intuitive sense of "understanding" but we don't have a good formalism around it, and this intuitive sense is easily tricked. But since we don't know how to formalize understanding, we are not yet capable of building AI which truly understands concepts (abstract and concrete) in the same way that birds and mammals do.

As a specific example, I suspect we are several decades away from AI being able to safely perform the duties of guide dogs for the blind, even assuming the robotics challenges are solved. The fundamental issue is that dogs seem to intuitively understand that blind people cannot see, and dogs can therefore react proactively to a wide variety of situations they would have never been trained on, and they can insist on disagreeing with blind people (intelligent disobedience) rather than being gaslit into thinking a dangerous crosswalk is actually safe. The approach of "it works well most of the time, but you gotta check its work" might fly for an LLM but humans need to trust seeing-eye dogs to make good decisions without human oversight.

In particular, being a seeing-eye AI seems much more difficult than fully autonomous driving, even considering that the time constraints are relaxed. Buildings are far more chaotic and unpredictable than streets.

[1] Note these concerns are not at all relevant for the research described in the article, where "learn" means "machine learning" and does not imply (or require) "understanding."

Jazz guitarist here, utterly disgusted by this technology: you are right on the money. The chorus is fine (though bland), but the verses are simply terrible - not just bad timing, there is a total lack of melodic "flow." Like almost all AI music I've heard, the melodies fit into one of two categories:

1) generic and lazy, but works musically due to Music Theory 101 (playing chord tones to the beat, cliched progressions, dumb rhythms)

2) arbitrary and nonmusical, not at all like a bad musician (more like a baby on a piano)

I will add that LLM hallucinations often blend into the text, and an art generator screwup might be hidden in the background. But when a generative music AI hits an off note in a distinctly nonhuman way, it's obvious and very jarring, even if it's only a handful of times in a long song.

There's no way the prompt actually works, though. LLMs are not able to reliably "preserve the overall meaning" of things unless they're doing direct technical translation. The problem is going to be even worse with original research, because the LLM will try to summarize according to old ideas from blog posts / etc in its training data, and not the new ideas in the original research. In general document summarization is one of the worst use cases for LLMs, both in terms of its reliability and the difficulty of finding errors - how would you know without reading the paper? I would be surprised if this prompt worked on a single honest[1] paper that was written after the LLM was pretrained.

The bit about translating LaTeX expressions into human-comprehensible math sentences is interesting and AFAIK should work on something like GPT-4. But that's just a case of technical translation. GPT-4 definitely cannot "rephrase the overall paper... simplifying along the way." GPT-4 can't even summarize corporate reports without screwing up facts and figures - why on earth would you try to use it to summarize new scientific research?

Stuff like this is why I'm so concerned about LLMs: this prompt doesn't work, and people using AI for this stuff is just automating ignorance. Very frustrating.

[1] I say "honest" because this prompt would probably do ok on stuff coming out of a paper mill - the problem is carefully stated original ideas. GPT tears original ideas to shreds.

Ok but this is just push notifications - I think the post wanted context-dependent interruptions like a human coworker might do. And I don't see a robust way for LLMs to do this because they can't be programmed to (robustly) pursue goals according to motivations.

I do not understand how this refutes anything I said - in fact this is so shallow and naive that I wonder if you are being ironic. If you're not being ironic... I suspect I will be unable to convinced you otherwise.

You are prompting an LLM to temporarily behave in a certain way. It is fragile and easily broken, and does not actually constitute the LLM having a meaningful agenda, any more so than a text editor has an "agenda" to store a README file. And ultimately this sort of prompting is just a trivial variation on this:

> I could see LLMs interrupting if you are typing something clearly false or against TOS. But that would require an LLM which reliably understands things are clearly false or against TOS and hence requires a solution to jailbreaking....so in 2024 I think it would just be an incredibly annoying chatbot.

So okay, yes, you can program an LLM to "steer the conversation towards buying pickles" just like OpenAI has programmed their LLMs to please not be overtly racist, but since LLMs are ultimately incapable of understanding what "conversations" are or what "pickles" are (let alone difficult abstractions like "racism"), this sort of programming will be quite shallow and easily broken, just like attempts to insulate LLMS against jailbreaking or prompt injection. I suspect if I kept talking to your LLM one of two things would happen:

1) It would completely forget about the pickle prompt and go back to being a generic chatbot

2) The interjection of "Bill's Pickle's Gourmet Pickles" would quickly become facile or annoying - the LLM is not actually intelligently reacting to the conversation and trying to "steer" things, it is just blindly repeating pickle-related sales verbiage.

Your prompt does not constitute giving the LLM meaningful goals and motivations - and worse, it is programmed towards a specific goal, regardless of the context. It is a shallow imitation of an agenda, and simply not the same thing of an animal having an agenda in the sense described by Saint Augustine[1]:

> Did I not, then, as I grew out of infancy, come next to boyhood, or rather did it not come to me and succeed my infancy? My infancy did not go away (for where would it go?). It was simply no longer present; and I was no longer an infant who could not speak, but now a chattering boy. I remember this, and I have since observed how I learned to speak. My elders did not teach me words by rote, as they taught me my letters afterward. But I myself, when I was unable to communicate all I wished to say to whomever I wished by means of whimperings and grunts and various gestures of my limbs (which I used to reinforce my demands), I myself repeated the sounds already stored in my memory by the mind which thou, O my God, hadst given me. When they called some thing by name and pointed it out while they spoke, I saw it and realized that the thing they wished to indicate was called by the name they then uttered....So it was that by frequently hearing words, in different phrases, I gradually identified the objects which the words stood for and, having formed my mouth to repeat these signs, I was thereby able to express my will.

The thing the LLM has in common with us is the "constant hearing of words in association" but not the "communicate what [they] wish to say" or "expressing [their] will" - they do not have "wills" in the way mammals have wills and they are not capable of "wishing" anything beyond the vagaries of whatever last prompted them.

[1] https://faculty.georgetown.edu/jod/augustine/conf.pdf

"Take control of the conversation"...and do what? Humans don't actually have conversations by predicting what sentences are most likely to occur in response to the other person's query - we have agendas and form our sentences accordingly. So if we interrupt another person speaking, it's because we have a specific, often personal reason to do so: perhaps we want to steer the topic of conversation to something we are interested in; perhaps somebody is suggesting a clearly bad idea; we might correct misinformation; take irrational personal offense; and so on. Interruptions are by design antagonistic, and among humans they involve a conflict of agendas.

But LLMs don't have any agenda whatsoever - they are not capable of having goals or motivations. So why are they interrupting? Are they reading your mind and understanding your goals before you even finish typing them? It's hard to see an LLM having a coherent way to interrupt based purely on a probabilistic view of language.

It would be very annoying if a human constantly interrupted you because they were "aligned with your agenda" and thought they were being helpful. LLMs would probably be much worse, even if they were able to reliably infer what you wanted. For an LLM to be useful, you kind of have to coax it along and filter out a lot of empty verbiage - it seems downright counterproductive to have that verbiage blasted at you by a chatbot that interrupts your typing.

I could see LLMs interrupting if you are typing something clearly false or against TOS. But that would require an LLM which reliably understands things are clearly false or against TOS and hence requires a solution to jailbreaking....so in 2024 I think it would just be an incredibly annoying chatbot. In general I think any interruption behavior would be artificially programmed to make the LLM seem "realistic," and it won't work.

Some of that is avoidable schism. But some of it is a natural consequence of Python by itself not being quite sufficient to solve the real-world problems people are throwing at it - in particular stuff like Conda is very useful for numeric Python since it ultimately abstracts away a lot of C/Perl/etc. which downstream Python packages depend on, but which pip and venv/etc by themselves aren't designed to handle.

> Mollaret seemed to think that, because crows are smart, their behavior should be predictable and programmable—even if his own behavior wasn’t. He was treating ecology like a subset of mechanics, as though the crows themselves could be turned into cogs in a machine.

In particular it's hard to get a crow to do tedious, boring chores for some almonds or whatever when they know there's relatively fresh pizza in a dumpster 2 blocks down the street.

IANAL[1] but is it possible that "vik" meant "home" or something as long as 20,000 years ago, before people even populated the Americas, and that this word persisted throughout the development of American and North Germanic languages?

[1] linguist :)

> Converse curiously; don't cross-examine. Edit out swipes.

> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize.

> Eschew flamebait. Avoid generic tangents.

In particular you seem to be responding to a strawman, since nowhere did I say C error handling was better than structured exceptions. The parent asked what was functionally different between laziness around checking error codes versus laziness around try/catching.

> Writing code with the occasional try/catch block isn't too different from writing C and not checking error conditions

I think a critical difference is that in C the program is more liable to simply crash if errors aren't correctly handled, whereas in Java/Python/etc the program can just log a stack trace and keep on truckin', even if the bug is actually quite severe. In some cases a crash is preferable - e.g. if something goes wrong in a text editor while saving data, it's a lot better for the user if the program crashes versus the alternative where the editor runs as normal but saving doesn't work. Crashes in C also bring more urgency for developers to actually fix the bug compared to a try/catch in Python that simply buries it "until I get a chance to debug it properly." (But crashing also leads to a lot of frustration when the error wasn't that important and the C program should have just kept going.)

The reason it's not that impressive is a lot of us are not as easily fooled as we might have been back in 2022!

As someone who also went to UWaterloo for quantum computing (MMath, not PhD, and I don't work in the field), let's look at the tweet in detail:

- in the first image, Claude's output is almost all useless LLM mush! Points #1, #3, #4, #5, #6, and #7 are uselessly vague and at best tangentially related to the problem. Only #2 has anything of interest, and even then it's just "come up with a quantum version of a classical algorithm," which is hardly insightful. Claude's output here just straight-up sucks!

- in the second image, "Initialization" is useless, "Quantum State Measurement" is useless, "Momentum Refreshment" is useless (and might be wrong in this context), "Iterations" is useless and misleading, and "Sample Collection" is useless.

- There is some real meat in the "Quantum Leapfrog Integration" section, but the entire last bullet point (starting with "The quantum leapfrog integrator consists of") is a completely useless statement which applies to every quantum algorithm.

There is almost nothing actionable for a quantum computing researcher here, except maybe as a brainstorming aid. And of course the giant pile of LLM mush is evidence that Claude doesn’t know what a “quantum computer” is or what an “algorithm” is, it’s just a fancy chatbot. I am not at all impressed by any of this.

A very important result from this study is that GPT-4 is in many ways worse than GPT-3.5, which was worse than GPT-3. More RLHF makes LLMs hide the overt racism better and leads to a false sense of security ("thank God GPT finally stopped using n-bombs") but they are learning more covert racism from their giant pile of badly curated training data, and this covert bias is not being adequately addressed in RLHF.

Any company using LLMs for hiring decisions needs to be investigated by the feds.

Speaking as a tech worker who has been formally diagnosed with schizophrenia: fuck off.

I am aware you are doing the "modest proposal" bit. Still: fuck off. People like you are why I never ask for reasonable accommodations for my disability, even when I am having serious health problems.

No, I mean in terms of raw computational power: 300Hz * 80 billion cores. Again the architecture does not work well for churning through a giant array of floating point arithmetic since 300Hz is a limiting factor for that specific type of computation. But that's not the kind of problem the human brain was designed to solve.

Neurons aren't anything like transistors: they're CPU cores, and each individual neuron in a mammalian cerebral cortex is the equivalent of a deep artificial neural network: https://www.quantamagazine.org/how-computationally-complex-i...

In particular the human brain is an 80-billion core processor with each core running at 300Hz. Its architecture is not so good at FLOPs and other things we typically use computers for, but overall human brains dramatically outperform anything in 2024 except the world's fastest supercomputer.

> They were different for sure, but the fact that we regularly mated with and formed families with Neanderthals and their first few generation generations means to me that the difference wasn’t that much.

To be clear there's only genetic evidence of admixture, not any evidence that humans and Neanderthals actually coexisted as families. Mating between humans and Neanderthals definitely wasn't "regular" since there's not actually very much neanderthal DNA in modern humans. Maybe we had an enormous number of infertile children, but I don't think human hunter-gatherers would survive that since rearing and raising a child is so costly.

In particular there's very good reason to suspect that we didn't form families with Neanderthals in the way you are suggesting[1]:

> No evidence of Neanderthal mitochondrial DNA has been found in modern humans. This suggests that successful Neanderthal admixture happened in pairings with Neanderthal males and modern human females. Possible hypotheses are that Neanderthal mitochondrial DNA had detrimental mutations that led to the extinction of carriers, that the hybrid offspring of Neanderthal mothers were raised in Neanderthal groups and became extinct with them, or that female Neanderthals and male Sapiens did not produce fertile offspring. [...] As shown in an interbreeding model produced by Neves and Serva (2012), the Neanderthal admixture in modern humans may have been caused by a very low rate of interbreeding between modern humans and Neanderthals, with the exchange of one pair of individuals between the two populations in about every 77 generations.

Even if it was 1 pair in 5 generations, that's simply not high enough to suggest a strong common humanity based on admixture alone. Pulling this all together, I don't think there's any real evidence that humans and Neanderthals co-parented or had any other real integration between their societies. There was just occasional admixture.

I am sure there's a paper somewhere but I've never read anyone actually rule out what is IMO the simplest explanation: Neanderthal men occasionally raped modern human women and the child was raised among modern humans (the reverse situation explains why children with Neanderthal mothers and modern human fathers would have been raised among Neanderthals).

[1] https://en.wikipedia.org/wiki/Interbreeding_between_archaic_...

I mentioned this in an HN comment earlier this week - be careful about the "Neanderthals had bigger brains than us" stuff. A big chunk of that difference is in the occipital lobe, which is involved in visual processing. And the likely reason for that is that they had significantly larger eyes than modern humans: Larger eyes = more pixels = more real-time visual processing demands (assuming Neanderthals had the same visual acuity as modern humans) = bigger chunk of brain, since evolution is not gonna easily minimize the neurons themselves or find a more efficient algorithm. But on the other hand, their brains being comparable in size to ours is a good reason to think their intelligence was basically comparable to ours.

FWIW this study suggests that Neanderthals and modern humans might have had very different approaches to language: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4766443/ Neanderthals and modern humans both evolved to make sophisticated vocalizations, but it's possible that the Neanderthals had 10000 words yet only a few ways to join them in simple sentences. (E.g. what if Neanderthals couldn't ask questions, only state disagreements?) In particular we know that humans can have local brain damage leading to near-total disability in using language while still being able to comprehend language, so you could see ancient hominids being "as smart as modern humans" in terms of general problem-solving - building tools, administering medicines and performing surgeries, long-term planning around food - but having profound blocks in language that prevented them from forming larger societies. Neanderthals tended to live in small family groups, whereas almost every modern human hunter-gatherer lives in a much larger tribe (absent war or disaster).

> It's not clear the Earth can even do that anymore.

This is a good point, but the event wouldn't need to be more volcanic so much as that volcanism would need to ignite more carbon-containing materials - naively I would expect there to me more "fuel" in the Earth than there was 300m years ago.

To be clear I don't think anthropogenic climate change will cause human extinction. But a terrible series of volcanos could absolutely turn Earth into a colder Venus for, say, a couple hundred years - pelagic vertebrates and certain microbes would be ok, but I am not sure humanity would survive more than a few generations, even in underground sci-fi bunkers.

https://en.wikipedia.org/wiki/Permian–Triassic_extinction_ev...