D-Machine

> The proposed SoC was extensively validated on two intracranial electroencephalography (iEEG) datasets related to psychiatric disorders, achieving 80.9%/74.8% sensitivity and 74.2%/73.1% specificity on the multi-source interference task (MSIT) and the emotional conflict resolution (ECR) task, respectively. We further evaluated the SoC on the public DEAP dataset for electroencephalography (EEG)-based emotion recognition, achieving 70.5%/72.5% sensitivity and 69.8%/66.3% specificity for the arousal and valence states, respectively. [https://ieeexplore.ieee.org/document/11685197]

I dunno, really need to see the comparative performance of other classifiers and the base rates in these datasets to know whether these performance metrics are anything we should care about, and whether e.g. a classifier on simple verbal probes to the subjects might simply yield better predictions than going through the EEG signals. You can do some really interesting EEG biofeedback stuff, so maybe this is useful, but I would think the full EEG head-gear would also provide much richer and more useful information than a measurement based on a single chip. Presumably this would be for psychiatric disorders or brain injuries where the core dysfunction we know is limited to a very specific region(s).

Yup, I would say violent agreement is right! 100% agree with everything you are saying here.

> LLMs are showing us that a lot of what we think of as cognition is actually part of the elaborate symbol system which is Language

This is entirely speculative and highly disputed, even if you are doing the more sophisticated thing here and by Language you mean something like mentalese.

Much cognition is embodied and makes use of visual, tactile, auditory, kinaesthetic, and other sense modality representations. "Thought is mostly a Language" has been a wordcel kind of argument with little empirical or philosophical support for a couple of decades now, IMO. Language is how we communicate thought, not primarily how we think.

Not that this refutes your main point: LLMs are trained almost entirely on (written) language, and since language is only a single, limited aspect of real cognition, this makes them so annoying and alien to read.

The emotion / cognition distinction is somewhat poor and often breaks down, because you can't really have one without the other (different mental states just involve differing degrees of both, phenomenologically). They aren't really causally separable, but are only really phenemonologically separable (cognition and emotion have different feels / textures, but almost always co-occur).

Most importantly, you rarely have "cognition-free" emotional states, because emotions always involve some mental model or representation of the self / world / self and world, i.e. they involve a cognitive state. Only in very extreme cases (drugs, perhaps some cases of persistent depression, etc) might you be able to have emotions that are largely uncorrelated with any kind of particular cognition.

Likewise, the same is almost certainly true for cognitive states, in that we tend to think about / model things because we need to evaluate those things. Evaluation too seems to rely heavily on emotional processes, in that damage to emotional processing tends to make rational / logical decision-making severely impaired (e.g. Damasio's work in the 90s, or https://pmc.ncbi.nlm.nih.gov/articles/PMC2396787/). Regardless, you can't remove evaluation / valuation from most thinking, and that is fundamentally linked to emotion.

Doesn't mean you couldn't potentially build a machine / thing that detects mental states, mind you. But it likely won't be very useful if it is just using simple folk-psychological terminology and concepts (or using such concepts will likely not result in much insights into the machine outputs).

> The 'b' sound is a plosive too, by the way. It's voiced, vs the voiceless 'k'.

Yeah this I know, by "bouba" being opposite I meant the waveform.

Really the phonetic terms here are not actually relevant and have betrayed me. It's more that the 'k' sound is "more [ex]plosive" in that it seems to build up more pressure and turbulence (probably as you say, because the consonant is in the rear, so pressure builds and bounces around in the mouth more, whereas the "b" just mostly blasts out the front more quickly). And the "k" surely has a more peaky / spikey waveform in the audible range.

> Well /b/ is also a plosive. :) Phonetics is out to sabotage your meaning.

Lol, true.

Even the abstract answers some of GPs questions. What is wrong with some of the posters here?

I was taught it was "BOW-ba" or "BOH-ba", but AI actually linked a number of studies that seems to confirm it is supposed to be and usually is "BOO-ba" (i.e. bulging eyes Pepe) in experiments. I don't personally think either pronunciation would change the results though.

Either similar underlying processing machinery (due to auditory and visual processing evolving from simpler and more interconnected processing circuits) or because when thinking of frequencies of waveforms in light/sound, visual sharpness and auditory sharpness really are the same thing (so even if they didn't co-evolve, it can be more efficient or useful to have some single underlying representation waveform representation that represents these things similarly).

> I do not think this is realible experiment. Dogs are very sensitive to non verbal cues from their owner

If you had actually read the article, they explicitly tested for and controlled this...

Oops, you are right! Thank you for correcting. I checked before posting that some "k" sounds are indeed fricatives, but they definitely aren't in "kiki".

It isn't trivially obvious, no, but the "k" sound is a fricative (EDIT: oops, nope, plosive - see response to this comment), i.e. the tongue rapidly moves close to the palate and then away, creating turbulence, and sounds with a sharp peak and amplitude. I.e. the waveform of a "k" or "kiki" sound looks more spiky. The opposite is the case for the "bouba" sound.

It is a bit mysterious why animals would all map e.g. sharp auditory waveforms to sharper / spiky lines (kiki figure), and then smoother sounds to smoother lineforms (bouba figure), but, reasonably, these senses evolved together and used far more shared underlying processing machinery than they do presently. After all, both light and sound reach us in waves and both carry energy.

Some primitive mush-blob perhaps just started processing all incoming energy waves together, differentiating e.g. smoother vs. more staccatto waves before making the distinction between e.g. sound vs. light. Or I dunno, but IMO it makes more sense when you consider evolution, and don't just look at processing now in isolation from the likely development of these senses.

Precisely, the mathematical insight is the essence of the post here, you can't simplify it or change it to some other form without destroying the point and the beauty of the insight. And yes, the math here is trivial. The hubris, anti-intellectualism, and lack of curiosity of these complaints, on HN of all places, is really shocking.

> If the blog had pseudo-code instead of 18th century scribbles

E for expectation, blackboard bold, double-bar norms and indexed max/min are all early to mid 20th century notation, making them thoroughly modern. Only summation notation dates since the eighteenth century, and its relative stability over that period is in fact rather a strength and not a weakness. Summation notation is also nearly universal in any quantitative field, and far, far more concise than the equivalent looping C-style pseudocode or functional-style pseudocode.

> You're also connecting the syntax to understanding of the concepts which is the cherry on top of the math snobbery in this thread

Because that is in fact a big part of how it works. Good notation takes a lot of time and thought to develop, and the notation used in the post is far, far more concise and packs far more information into a tiny amount of space than pseudocode or other programming conventions. Much of math is abstraction (compacting complex ideas into specific symbols or concepts, and then coming up with simple rules to allow their manipulation), and is possible in large part only because of good notation like this. See some of Terry Tao's thoughts on the subject below.

The notation here is really also all first-year introductory notation, and incredibly simple. Most CS or computer engineering students should have seen it, and anyone that thinks they can learn anything non-trivial at all about neural networks by avoiding such basic notation is simply not a serious person at all.

EDIT: And it is a mistake to imply pseudocode is generally superior or easier to understand, because pseudocode is often too precise, and implies more structure and relations and implementation details than are actually needed or important. Meaning it actually often communicates the wrong thing compared to good notation.

https://terrytao.wordpress.com/advice-on-writing-papers/use-...

https://mathoverflow.net/questions/366070/what-are-the-benef...

See my comments and links below, you are not wrong, IMO this is not actually a great analogy for understanding LLMs (other than as an explanation for why a trained one can so efficiently compress such a large amount of information).

https://news.ycombinator.com/item?id=50131741

Before getting too hung up on memory and seeing these as key value stores, or self-attention is suppression of interference, recognize that a lot of experts and people in these areas disagree with this analogy for fairly deep reasons (I link other resources and discussion here: https://news.ycombinator.com/item?id=50131741). It is still a useful analogy, but it is incomplete and weak in many ways when it comes to the actual details of training.

Also make sure to understand how self-attention layers are actually applied and the exact math and matrices involved. E.g. in PyTorch you are going to be doing sa(x, x, x), where x is the previous layer's output (essentially, embedded tokenization), and sa() does quite a few matrix multiplications. In cross-attention, you are doing sa(x, x, y), where y comes from a different part of the architecture.

Basically, read broadly here and don't get stuck on a single analogy / intuition for what these layers are doing. Focus on learning the actual math of the layer and the network first, and then look for analogies that help pump your intuition. But don't try too hard to force a single analogy onto the math.

EDIT: Also modern AI can provide useful insight into posts like this, I highly suggest using them when learning these things. E.g. https://chatgpt.com/share/6ad3aa3d-d110-83ea-a8c5-8653b34743...

Thats MathJax, you need to allow JS to load from cdn.jsdelivr.net. If doing that doesn't cause the math to render, you likely have some other issue (perhaps a very old / strange browser, a caching issue, or some other kind of blocker additionally preventing the JS from loading).

Yeah, the notation and math concepts also all just first-year calculus (or analysis, more accurately) and stats (and the latter being only E for expectation). Literally just summation notation, blackboard bold R for the real numbers, double bars for norm / distance, and then max/min with subscript for the quantity being minimized.

Embarrassing to complain about notation so simple, and, frankly, if you can't understand math concepts and notation this basic, you simply can't meaningfully understand modern neural networks: without the math, all you are doing is substituting loose analogies and metaphor for understanding. This is clearly an article for people that want more than that.

Yes, the main problem with this whole way of thinking about transformers and self-attention is that when you look at how they are actually implemented (i.e. what Q, K, and V are applied to at each layer, and consider the loss function during training), then the analogy to memory quickly starts to look tenuous and far-removed from what is actually happening.

Mainly, since self-attention layers operate on deeply-stacked representations of the outputs of previous layers, and since training has a loss function that doesn't make the updates work in the way the article says, the analogy to memory and lookup tables breaks down quite strongly. IMO, thinking of Q, K and V as memory or key-value lookup is thus not actually very helpful for understanding what these layers are really doing or why they work. Others agree (e.g. https://news.ycombinator.com/item?id=46535412), so don't feel bad if you think there is something hand-wavey about the lookup-table and memory explanations - it is only a weak analogy.

This has come up on HN before, and probably it is most accurate (but less intuitive, sadly) to see self-attention as a generalization of kernel methods (https://news.ycombinator.com/item?id=46535549). Broadly, self-attention is just one of the simplest ways to model similarities / correlations between all inputs (i.e. kernel method), but there are many, many other attention mechanisms. There are also many other ways to implement memory (e.g. LSTMs), and many other architectures that might in fact have even more memory capacity than transformers.

That being said, this is still truly an excellent article and it really provides some great intuitions for why transformers might work as well as they do. The softmax / exponentiation here functioning to increase memorization capacity by suppressing weak activations is super clear and interesting, and I really appreciated it. I suspect is not really be what is happening or is actually important about it, given other research (https://arxiv.org/html/2410.18613v1), and because much of what works in DL is usually more about matrix conditioning and keeping gradients well-behaved generally, rather than what the actual bare final math / weights in isolation from training would tend to suggest. But it does give a great intuition for why a trained transformer can have the capacity it does.

You haven't said anything of substance, because as I and other commenters here have pointed out, you clearly don't even understand Rice's theorem and why it isn't applicable here. You bring up formal verification with Lean on a thread and topic which is about model explainability. You should just stop posting entirely, you are polluting HN with irrelevant nonsense.

You can certainly use both X/Twitter and BlueSky (and Mastodon, for that matter), and I did for a while.

Just if you curate your Twitter feed, you gain very little extra from BlueSky or Mastodon, and if you choose only the latter two, you get a much more biased and limited set of views. (Twitter has plenty of crazies from all parts of the political spectrum, but BlueSky and Mastodon run mostly left, and have much, much smaller user bases). HN also is extremely limited in what it discusses, so it doesn't make up for what is missed by ignoring X either, IMO. X is still one of the most efficient general public squares by a long shot.

If you can't stand Elon and don't care about what you're missing on X though, I can fully understand just not bothering with it. You aren't missing much.

> Emergent means something along the lines of not easily predictable behavior from the behavior of the component parts.

Yes, partly. But it also refers to complexity arising from the component parts that requires additional tools (formulas, descriptions, etc) to meaningfully describe. Or at least, emergentism has always formally involved more than mere predictability from parts (https://plato.stanford.edu/entries/properties-emergent/). I am talking about weak emergentism here, so no need to get into metaphysics and mysticism.

> My claim is nothing in LLMs is emergent in the sense of being inexplicable from an ideal consideration of simple/linear aggregations of its neurons.

IMO this contradicts even your minimalist definition. Almost no one predicted simple LLMs would be where they were today, and you can't currently predict their behavior by looking at their weights.

Something doesn't cease to become emergent if you can roughly explain it by making references to its parts. We can broadly explain how human behaviours result from drives + sensory processing + associations and etc., but human beings are not very predictable in many cases, and also exhibit consistent behaviors that tend to be very hard to predict (or even explain) from basic knowledge of the brain, drives, and body. LLMs are no different, and what they are doing today couldn't be more clearly emergent, even by your definition.

Also, something doesn't cease to be emergent even if you can perfectly explain it with reference to the parts. E.g. attractors are emergent behaviour, even though they fall out of very simple formulas and so in a sense are completely due to their defining parts. But describing the shapes of e.g. the Mandelbrot set or the Lorentz Attractor requires more than the base formula that generated them. So they are classic examples of emergent phenomena. Likewise, a person's intelligence might one day be fully predictable from biological and genetic knowledge and measurements, but it wouldn't mean that intelligence stops being an emergent property that is still observable and useful to use itself in other models.

You can for example argue that an LLM has no emergent drives (https://xxcancel.com/jon_stokes/status/2111158233993797870#m), and that would be a coherent claim, under a certain definition of "drive".

But your argument here I cannot follow clearly, because LLM behavior is clearly emergent in any sense of the term, including yours. EDIT: Or basically, LLMs show weak emergence, but if you disagree with this, then it would seem to me you perhaps disagree with the concept of emergence at all.

Agreed.

> Emergent behaviors in terms of seemingly random combinations of features producing unpredictable but coherent/intentional output basically don't exist.

You'll need to elaborate here, because Navier-Stokes plus Mathocalypse reactions are extremely strong evidence for coherent emergent behavior coming out of these models.

> The only thing emergent in LLMs is degenerate state leading to a breakdown in tracking relevant features of the input and a reduction in quality of output.

This is so provably wrong with two seconds of Googling / AI-asking that you must be using some completely idiosyncratic definition of "emergent" here.

> I expect this has as much to do with our lack of precision in our tools rather than with the concept in principle.

Unfortunately in this case it is the exact opposite.

Nothing in this thread was about Lean.

Nothing you said responds to anything I said.

Rice's theorem (which you are clearly citing, but won't clearly mention, for some reason) has almost zero actual practical relevance to AI explainability.

You make low-quality and low-effort AI-denialism posts whenever you pattern match "all non-trivial semantic properties of programs are undecidable" to... anything you think matches the folk/common understanding of those terms, without considering how irrelevant these statements are to actual real-world applications and engineering.

You ask for links, but post nothing of substance and post no references or even mention relevant details for your own claims.

Also just curating your feed (i.e. judiciously using blocking and muting) and instead viewing just "Following" removes something like 99.9% of all common complaints re: Twitter, while still giving you access to a much less-biased (or easier to equally-bias through curation: e.g. you can't get an accurate view of RW ideology following only BlueSky posters) and far larger public square than anything else.

For responses, see the linked Twitter thread in other posts here, but also:

https://xxcancel.com/NeelNanda5/status/2110875974296654062#m (and the re-post on LessWrong: https://www.lesswrong.com/posts/x4kvabkF2Jk8GLZp8/linkpost-n...)

TL;DR:

    This is interesting work, but I don't think it's a significant update to the big picture of interpretability. [...] I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc.

> This seems like a classic model-description issue more than anything.

I agree.

> Modern researchers obviously want the goal of simplicity & composition, but to arrive at that, they build abstractions like transcoders and SAE's to get into their pet ontology (with the assumption that neurons themselves are garbage to understand). This obviously raises the question - why privilege your basis?

Agreed, and IMO the implicit reason for privileging here is supposed to be "consistency with existing human understandings and concepts", and this is why we get short pithy things like "V-shaped" and lots of pictures showing relatedness that we all can see and agree on. But what if the domain of inputs is NOT something we understand (e.g. the whole of mathematics, or something new or that we don't have any understanding of)?

I called this a validation problem in another comment here (https://news.ycombinator.com/item?id=50108184). I.e. if we apply whatever methods Nick is using to "explain" a model on a domain we don't understand (like a deep model trained only on fMRI and EEG data of the whole human brain in various contexts), we aren't going to be able to tell if that description is correct (or just better than any other description), because there is no simply visual / common-sense inspection of the description.

> The resulting interactions between said neurons may be incomprehensible, but the point is that this provides a layer of objectivity

Agreed, emergent behaviour is clearly the case and limits how useful understanding individual neurons can be. IMO this is especially the case because layers are (weighted) summing generally millions of contributions from previous neurons.

> But if he's right, then you do actually have this privileged base layer to work with, even if you don't know what the base layer means. That's pretty important, actually. Assuming this holds up in reality, ofc.

Yes, it not being "chaos all the way down" is good, actually. And this post shows there is indeed a good amount of order hidden away if you look at in the right manner.

Scanning back, I can see I was responding here was what I was worried was a mistaken assumption that a meaningful abstract / summary can always be written for truly complex and novel topics. Sometimes all it can be is a sales pitch for reading the details, and I thought the summaries here were good enough in this sense.

But I agree the author is obfuscating weak claims via length / complexity (and it may be as much for his own sake as for his readers'). The basic claim ('maybe AI will be more explainable/tractable than initially thought') is IMO intelligible, and not completely nonsensical, but also extremely vague and vibesy, which, technically, I must agree is in the same territory as nonsense, at least.

Based on our other exchanges here we mostly agree with the problems of the article I think.

> Neurons having narrow semantics means you can inspect what a neural network is "thinking" with confidence

Not really, because of emergent behaviours and also because there are still millions of neurons being summed (with variable weighting depending on inputs) at each layer.

> and even constrain its space of behaviors by activating/suppressing a small set of neurons that have very straightforward effects on its behavior.

You can influence behaviour by doing this kind of thing, yes, but it isn't straightforward at all, and is generally very messy and you get all sorts of unpleasant sides effects (e.g. abliterated models not performing as well).