Systems design 2: What we hope we know
apenwarr.ca
apenwarr.ca
> Any sufficiently advanced technology is indistinguishable from magic.
I had a lecturer who used to counter "if something ever seems like magic it means you're not asking enough questions."
That stuck with me, although the article goes on to counter it fairly well.
Anyone who spends enough time studying biochem will have an intuitive understanding / acceptance that biology emerges from chemistry, which emerges from physics… But there’s not really any “magical thinking” (as defined in the article) involved.
Emergence of cognitive processes and experiences in the brain is not really any different in this respect. If anything, it’s people who insist that human experiences are “special” and “just different, obviously” that are doing the magical thinking.
To me, "magic" means that a phenomenon is inherently inaccessible to naturalistic inquiry, not merely that such inquiry is temporarily stalled due to inattention, lack of the right insight, or insufficient technology.
Even just within "physics" there are probably multiple layers of emergence (a striking and popular example is particle-like behaviour arising from wave-like fields). There is "reasonable speculation" as to whether space, time, gravity might all be emergent rather than fundamental...
The author mentions Wolfram, but I think fumbles around for a reason why scientists were ambivalent about Wolfram's work. The reason is that clever and even surprising demonstrations of cellular automata have not yet yielded any connection to empirical science, thus it's premature to call it a "new kind of science." Not because scientists are opposed to computational models. Physics used up its supply of "closed form equation" models roughly a century ago, and has fully embraced computation. We even suffered through a few decades of grinding through computations by hand, waiting for computers to be invented.
If you stretch the definition of cellular automata a bit, you end up with models that divide regions into a grid of cells, with evolution rules that lend themselves to iterative computation. Examples include finite element analysis and climate modeling.
I'd wager the vast majority of laypeople cannot conceive how humans (or life) came to be without some sort of divine intervention. I certainly don't think anyone has the answer to that haha... But having this vocabulary, and "mere knowledge" of the existence of various examples of emergence can replace this incredulity with a more grounded appreciation of the inherent beauty and wonder of nature. I think the concept of emergence is profound in itself, because of how it can shift our perception of our own place in the world.
...knows about how LSD was used to discover DNA. Agreeing it possible is one thing, but to actually expect it? That is magical thinking.
I don't think putting random chemicals in your brain is all that different than putting a bunch of random coin-flips in a training function, so I don't think it's a terrible analogy either: The fact that something amazing comes out of that process is magic.
> I am uncomfortable with the assertion that somehow understanding emergence has anything to do with magic or magical thinking
That's common amongst scientists. Words matter, and replacing magic with "emergent" keeps it at arms length, and detachment is important to do science.
But I suggest you play along anyway, instead of pretending they meant "emergent" every time they say "magic" because words matter...
> If anything, it’s people who insist that human experiences are “special” and “just different, obviously” that are doing the magical thinking.
I hear you, but it is hard to operate on the assumption you're not special, after all, you're probably more successful than a lot of people, and I bet you think your hard work had something to do with it.
The science says this isn't true, of course, that the colour of your skin, the postcode you were born in, that the wealth and other genetics of your parents, are all greater predictors of success, and you probably know this, but if you're a human being you can't help but think that you have willpower and ethics, even if you don't believe in a soul.
Or another example: if you've ever looked at someone who behaved badly, and said you wouldn't have done that? Of course, if you really did have their life, their parents, their postcode, their skin, their abuse and traumas over the years, their joys and friends, stories of parties and life, and so on, well then you definitely would have, or you've got to believe you're special.
This kind of magical thinking has kept you alive- it's provided impulse to the bag of biology and chemistry to twitch and react its way through your life- something you've got to admit sounds a lot less fun when I use science-words to put it that way.
Is there some sort of allusion to the meritocracy trap or fundamental attribution bias? How does any of that relate to my comment? I'm just confused.
In any case I believe the connection you make between LSD and the discovery of DNA is apocryphal.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4813425/
“Would I have invented PCR if I hadn’t taken LSD? I seriously doubt it […] I could sit on a DNA molecule and watch the polymers go by. I learnt that partly on psychedelic drugs.”
> Is there some sort of allusion to the meritocracy trap or fundamental attribution bias? … I'm just confused.
I have no idea what led you to that idea and I’m very sorry to have confused you, what else can I say?
Is where OP perhaps got fundamental attribution error from.
> ...you're probably more successful than a lot of people, and I bet you think your hard work had something to do with it.
Is very similar to the myth of meritocracy some folks believe in.
---
In sum, you intend to point out that cognitive biases (or perhaps, cognitive dissonance) are also a form of magical thinking? Like OP, I find it hard to reconcile that with LSD/DNA discovery. In my view, biases mostly fool you; but they don't quite emerge out of an insight (magical thinking), as the article describes it.
This is a curious line - I have, and I personally have that reaction when I use it. It might even be the most magical technology I've ever encountered.
Literally in the next sentence the author also remarks that he finds it odd:
> “That makes me curious.”
He finds it curious because he also finds the technology is magical.
Eventually he concludes:
> “Let's say you have a Large Language Model AI, trained on all the text in the world. We understand the mechanisms (engineering) but the outcomes are surprising and emergent (magic). Its job is to repeatedly predict the next word that would come if a person wrote it – a form of negative latency, or non-causality. From this, for reasons we don't really understand, it can say an impressive variety of unexpectedly meaningful things.”
> “When you have all the text in the world memorized, that means you have access to all the insights that have ever been written. You need only recognize them, and have a good idea of what the reader knows already, and you can produce insights – things the reader has never heard before – on demand. You can get an A+ from my English teacher. Can our AI do that? I don't know. Certainly sometimes, maybe not reliably yet. Any time it does, it will feel like magic.”
No, "magical thinking" has a particular meaning, and it's not that: https://en.wikipedia.org/wiki/Magical_thinking
Mind you, people often use the phrase in a pretty loose way as a way of dismissing an argument, but it does seem to be based on the technical use, and not the meaning Apenwarr puts forth.
However I still find the argument very unpersuasive, and the usual meaning of “magical thinking” makes me feel uncomfortable too.
> On the other hand, people who believe AIs and specifically LLMs possess "intelligence" are often accused of "magical thinking."
If software engineering revolves around writing software complex enough to have bugs not understood, or accepting memory safety errors as a natural byproduct of programming (rather than programming mistakes to be exterminated on sight, and made impossible by design unless it prevents writing useful programs), I'd be more effective and happier practicing applied math than engineering.
When you build systems at scale you need to worry about these things.
Programming language API boundaries don't make real-time performance guarantees, not even in Rust. We hope to get typical performance. In a mathematical context (for example, proving programs correct), you often ignore these things and worry only about whether the program will give you the right answer, eventually, when the hardware works.
Lovely idea towards the bottom too,
> Or maybe we need to draw a distinction between producing insight and recognizing insight.
I personally identify with this & see it as super core a distinction. I'm reasonably bright & can find some clever ways of doing things even if I haven't any learnings here. But like 90% of what I love & appreciate about self is a resourcefulness & ability to recognize & integrate insight.
Maybe the idea of magic shouldn’t keep being referenced, we should refrain from comparing and just apply equal respect to all beings and phenomena and just admit some things are, and always will be mysterious and move on with life ?
"Summarization can be done with little to no insight at all. Perhaps to do it better requires some insight."
By this standard caching requires little to no insight at all. Perhaps some predictors require insight.