Hacker News Reader

Comment by worstspotgain | Hacker News Reader

Birds inspired planes. Later, aerodynamics fed back into ornithology. I've been waiting for LLMs to be evaluated as a model of human thought. Complexity and scale have held neuroscience back. It's nowhere close to building a high-level brain model up from biological primitives. Like ornithology, it could use some feedback.

All arguments about AGI aside, a machine was built that writes like a human. Its design is very un-biological in places, so it's tempting to dismiss it. Why not see how deep the rabbit hole goes?

Like all conjectures it just might surprise us. For one, the intuition that language is central to thought predates LLMs, but it's certainly consistent with it.

View on HN

> a machine was built that writes like a human

But the real hero here is not the LLM, but the training set. It took ages to collect all the knowledge, ideas and methods we put in books. It cost a lot of human effort to provide the data. Without the data we would have nothing. Without GPT we could use RWKV, Mamba, S4, etc and still get similar results. It's the data not the model.

> the intuition that language is central to thought predates LLMs

Language carries AI and humans. The same distribution of language can be the software running in our brains and in LLMs. I think humans act like conditional language models with multi modality and actions. We use language to plan and solve our problem, work together and learn (a lot) from others.

Language itself is an evolutionary system and a self replicator. Its speed is much faster than biology. We've been on the language exponential for millennia, but just now hit the critical mass for LLMs to be possible.

It's not so important that GPT-4 is a 2T weights model, what matters is that it was trained on 13T tokens of human experience and it now "writes like a human". Does that mean humans also learn the same skills GPT-4 has learned from its training set mostly by language as well?

> But the real hero here is not the LLM, but the training set.

And in the case of windmills the hero is the wind. But the mill is still a fantastic achievement.

But can mills easily move? No, people can, so people are still viable.

>But can mills easily move?

Yes, that's their whole thing: moving when the air blows.

But can they relocate?

And if you rearrange the letters in “mill”? “I, LLM.”

qed

Correlation checks out.

Indeed, they are mostly parroting that knowledge with some reasoning from combinations of those patterns. That counts for something. It’s not what we do, though. Or not all we do.

Humans can just sit around aimlessly toying with stuff, reading books, etc. They’ll figure out some of these patterns on their own. Whereas, we have to give these things a ton of highly-curated, pre-processed data made by human minds of all kinds. Then, it’s usually 800GB-4TB for the good ones. They’re appear to be not in our league yet as learning machines.

We’ll be able to assess it better as multimodal models come online. We can train them like infants, then children, on books, random observations through cameras, TV, people reading to them, supervised feedback… all the stuff we do with humans. Then see if and how they match up in performance.

> Later, aerodynamics fed back into ornithology.

You mean to imply that birds have learned from jet fighter designs?

I fail to understand the point you're making.

Concepts from the mechanics of flight (e.g. lift and drag) have helped ornithologists understand birds better. Birds themselves did not learn much.

I believe the point is that we understand the birds' flight better applying the principles we learnt designing and flying airplanes. Similarly, we can learn more about human brain by applying things we learn from building the ANN back to the study of humans (anthropology in general, maybe neuroscience and psychology )

He didn't say it fed back into birds, but into the study of birds.

Knowledge learned from airplane propeller development, as well as from jet plane wing aerodynamics, has had a great impact on the design of wind turbine blades, which are now affecting the evolution of birds.

Why assume this is sensible? Aerodynamics can help discover general principles that apply to both birds and planes or whatever else. I don't see how this holds for LLMs and brains. The similarity between LLMs and brains is superficial.

Besides, with human beings, we have a host of philosophical problems that undermine the neuroscientific presumption that a mechanistic and closed view of the brain can account for mental activity entirely, like the problem of intentionality.

>I don't see how this holds for LLMs and brains. The similarity between LLMs and brains is superficial.

It's really not any more superficial than planes and bird flight.

Reply on news.ycombinator.com