Hacker News Reader

Comment by mkarliner | Hacker News Reader

I think velcro is the canonical example of this.

https://blog.karliner.net/posts/chatgpt-and-the-velcro-test/

> It will never, ever, come up with Velcro. If a concept does not exist on the Internet, or the corpus of training material, LLM’s will not invent it.

So this guy made up an imaginary test, provided a completely unjustified result to it- which casually confirms his ideas- and he's so proud he was right! What a good test he invented!

Not to mention that this test was created in 2023, when the best model we had was GPT 3.5!

I think if an LLM had read any discussions on how best to remove burrs from pet coats then it might eventually get to Velcro - I believe the inventor of Velcro got the idea from burrs?

> "I believe the inventor of Velcro got the idea from burrs?"

I think there's a provocative human-intelligence analog to be found there: "humans will never discover anything that doesn't already exist somewhere in nature. We can only recombine concepts we've seen—never invent truly new ones".

It's said that mathematics is "unreasonably effective" at describing the natural world. But what about the other way around? What if mathematics is unreasonably ineffective at escaping the convex-hull of concepts and patterns found in nature?

I'm working on a piece inspired in large part by William Thurston's "On proof and progress in mathematics"[1]. Should publish it next week.

Part of why math is effective is that its extraordinarily compressible. And you can take those ideas and apply them elsewhere (composition works well). The natural world has a lot of detail and is therefore hard to compress. That's why it's a good source of new ideas. Consider the relation between the McCulloch–Pitts neuron (MCP lol)[2], developed to model biological spiking neural networks, and an inspiration for modern artificial neural networks.

[1] https://arxiv.org/abs/math/9404236 [2] https://en.wikipedia.org/wiki/Artificial_neuron

> What if mathematics is unreasonably ineffective at escaping the convex-hull of concepts and patterns found in nature?

Interesting. My intuition here is quite the opposite, that math is vastly overpowered for physics. I don't think we invest a lot of work in figuring out math that isn't useful to describe the natural world.

So I though about "we can only recombine concepts that exist in nature - never invent truly new ones".

The best counterexample I could come up with was lasers. Yeah, all the pieces are there in quantum mechanics, but so far as I know, nothing in nature lases.

Semiconductor chips. Again, yes, all the physics is there, but you're going to be reaching pretty far to find something in nature that's like a CPU chip.

Dyson spheres.

Interrupt vector tables.

Riemannian geometry.

Databases.

I'm sure that people can poke holes in some of these (with varying degrees of far-fetchedness as to how directly the same concept is in nature). But I think there are enough counterexamples, where nature's "prior art" is indirect enough, to make the absolute version of the statement false.

"Humans rarely discover something that doesn't already exist somewhere in nature"? Yeah, that's probably true.

The wheel and axle is an often cited example. It was a big surprise when a bacterium was discovered with a wheel-like connection to its flagellum.

https://en.wikipedia.org/wiki/Rotating_locomotion_in_living_...

True. That mechanism is there.

And yet, whoever invented the wheel and axle, I'm pretty sure they didn't do so based on the bacterium's flagellum.

So, yes, the concept existed in nature. But it wasn't the inspiration for the human invention in any way.

Or if it ran a couple million soft-body simulations with the goal of creating a reusable temporary link between certain kinds of materials, I think it would probably get there. Just another argument from lack of imagination on the part of the author.

ya - I think my intuitions around velcro are firmly intertwined with my meat-space actuators and perceptual equipment.

In other words, I experience high-resolution multi-modal qualia all day and it's really noisy and I think a lot of new ideas and concepts come from obsessively exploring a small subset of those details. Maybe that's just me posthoc rationalizing my phd, though.

That's a bold claim. It was almost certainly true for models in 2023, when it was written. I think that the recent spate of mathematical discoveries at least significantly narrow the implications for even current models (one can argue that they are only re-mixing existing mathematical concepts to solve problems that humans initially generated, but in my opinion, at some point you are really narrowing what it means for an idea to be "novel"). And I would seriously consider taking odds against it for models within the next 3 years (since that's how long it's been since it was published).

We are basically already at the point where if what current frontier models can do isn't truly "novel", then the vast majority of humans have never had a novel idea in their lives. My three years from now guess is that if whatever it is that frontier models can do at that point isn't "novel", then no human ever will have had a novel idea.

LLMs assign non-zero probability to every output (unless you you sampling modifiers like top-k), so if you sampled enough times the model would definitely come up with velcro. It's the same as the proverbial monkey with a typewriter.

The real question is whether it would happen in a practical timeframe.

> The real question is whether it would happen in a practical timeframe.

How are we defining a practical timeframe? How many billions of man-years did it take for a human to come up with velcro? In a world without velcro I could easily imagine an llm proposing the idea long before a pure human team.

This is the same thing as "but AI can only repeat what it already saw" but the goalpost moved again.

Yann LeCun is right, sometimes

I dont read the argument as specifically AI being bad at the physical world but more that they can't be creative, physical world or not

It's ironic looking at a recent post on the same blog and immediately noticing the claudisms https://blog.karliner.net/posts/thingstudio-common-ground/

edit: clarification: the content itself is fine, in fact I'm curious!

Ok my bad. In my defence, it was the first time I ever asked an LLM to help writing something.

Reply on news.ycombinator.com