Comment by jsenn | Hacker News Reader

Comment by jsenn | Hacker News Reader

Parent Full thread

Sure, but the type of "language" that includes HTML and C++ is very different from the type of "language" that includes English and French. Chomsky's point is that there's something special about human brains that makes it very easy for them to learn English and French, even with very sparse and poorly-defined inputs, but doesn't necessarily help them learn to produce other types of structured sequences. For example, a baby raised next to a forest will effortlessly learn to speak their parents' native language (you couldn't stop them from doing that!) but won't learn to produce the birdsong they hear coming from the forest. This indicates that there's something special about our brains that leads us to produce English and not birdsong.

Similarly, it's true that some humans can, with lots of cognitive effort, produce HTML and C++, but some can't. Even the ones that can don't do it the same way that they produce English or French.

Orphaned humans raised by animals can never learn to speak natural languages either. But yeah they won't produce birdsong. There's no utility to that. I guess it's a matter of environment. And btw for me writing HTML is effortless, but then I've spent a lot of time around other programmers.

> But yeah they won't produce birdsong. There's no utility to that. I guess it's a matter of environment.

This is the crux of the issue. GPT-3 would happily learn birdsong instead of human language, just like it has learned to produce snippets of code or a sequence of chess moves or various other things found in its training data. For that reason, it's not by itself useful as a model of human cognition. Which is not to say it isn't interesting, or that studying LLMs won't lead to interesting insights into the human mind--I suspect it will!

Interesting point. The fact the GPT-3's training data doesn't have samples of birdsong in it is down to the OpenAI engineers not feeling it's important to put any in. So it's still limited by human cognition in that way. Maybe analysing what wasn't put in the training data would yield insights into the human mind as well.

Reply on news.ycombinator.com