Hacker News Reader

Comment by tripplyons | Hacker News Reader

I think that most human learning comes from years of sensory input. Why should we expect a machine to generalize well without any background?

View on HN

Newborns (and certainly toddlers) seem to understand the underlying concepts for these things when it comes to visual/hepatic object identification and "folk physics":

  A short list of abilities that cannot be performed by CompressARC includes:

  Assigning two colors to each other (see puzzle 0d3d703e)
  Repeating an operation in series many times (see puzzle 0a938d79)
  Counting/numbers (see puzzle ce9e57f2)
  Translation, rotation, reflections, rescaling, image duplication (see puzzles 0e206a2e, 5ad4f10b, and 2bcee788)
  Detecting topological properties such as connectivity (see puzzle 7b6016b9)

Note: I am not saying newborns can solve the corresponding ARC problems! The point is there is a lot of evidence that many of the concepts ARC-AGI is (allegedly) measuring are innate in humans, and maybe most animals; e.g. cockroaches can quickly identify connected/disconnected components when it comes to pathfinding. Again, not saying cockroaches can solve ARC :) OTOH even if orcas were smarter than humans they would struggle with ARC - it would be way too baffling and obtuse if your culture doesn't have the concept of written standardized tests. (I was solving state-mandated ARCish problems since elementary school.) This also applies to hunter-gatherers, and note the converse: if you plopped me down among the Khoisan in the Kalahari, they would think I was an ignorant moron. But it makes as much sense scientifically to say "human-level intelligence" entails "human-level hunter-gathering" instead of "human-level IQ problems."

> there is a lot of evidence that many of the concepts ARC-AGI is (allegedly) measuring are innate in humans

I'd argue that "innate" here still includes a brain structure/nervous system that evolved on 3.5 billion years worth of data. Extensive pre-training of one kind or another currently seems the best way to achieve generality.

Sure and LLMs ain’t nothing of this sort. While they’re an incredible feat in technology, they’re just a building block for intelligence, an important building block I’d say.

So all billons spent finding tricks and architectures that perfom well haven't resulted in any durable structures in contemporary LLMs?

Each new training from scratch is a perfect blank slate and the only thing ensuring words come out is the size of the corpus?

> Each new training from scratch is a perfect blank slate [...]?

I don't think training runs are done entirely from scratch.

Most training runs in practice will start from some pretrained weights or distill an existing model - taking some model pretrained on ImageNet or Common Crawl and fine-tuning it to a specific task.

But even when the weights are randomly initialized, the hyperparameters and architectural choices (skip connections, attention, ...) will have been copied from previous models/papers by what performed well empirically, sometimes also based on trying to transfer our own intuition (like stacking convolutional layers as a rough approximation of our visual system), and possibly refined/mutated through some grid search/neural architecture search on data.

Newborn brains aren't blank, they are complex beyond our current ability to understand. All mammals are born with a shocking amount of instinctual knowledge built right into their genome.

All organisms are born pre-trained because if you can't hide or survive the moment you're born, you get eaten.

> if you can't hide or survive the moment you're born, you get eaten.

uhhh... no, most newborns can't "hide or survive the moment they're born", no matter the species. I'm sure there are a few examples, but I seriously doubt it's the norm.

Many species survive by reproducing en masse, where it takes many (sometimes thousands of) eaten offspring for one to survive to adulthood.

In humans at least, they lack the motor control to attempt to hide , gather, or hunt obviously. But they do plenty of other stuff instinctively. With my latest, we learned that infants are inherently potty trained (defecation) if you pay attention to the cues…and I was surprised to find that that was true, baby communicates the need to go and knows what’s happening without any training at all. Amazed to have almost zero soiled diapers at one month.

Makes sense though, I’m pretty sure mammals don’t do well with the insects and diseases that come with waste saturated bed.

The point clearly still stands: every species on the planet has a long list of attributes and behaviors directly attributable to evolution and “pretraining.” And many, many more based on education (the things a lioness teaches her cubs.)

I’m not sure we would call anyone intelligent today if they had no education. Intelligence relies on building blocks that are only learned, and the “masters” of certain fields are drawing on decades and decades of learnings about their direct experience.

So our best examples of intelligence include experience, training, knowledge, evolutionary factors, what have you — so we probably need to draw on that to create a general intelligence. How can we expect to have an intelligence in a certain field if it hasn’t spent a lot of time “ruminating on”/experiencing/learning about/practicing/evolving/whatever, on those types of problems?

Please do some research into developmental psychology. Babies are far, far more sophisticated than you seem to believe.

Have you ever seen a human baby attempt to hide upon birth? No? Thought so.

Please have a baby and report back first-hand observations. Yes, they're far, far more sophisticated than most (all?) humans can comprehend, but they're also completely incapable for multiple months after birth. This isn't unexpected, human babies are born at what would still be mid-late gestation in almost any other mammal.

That quote about how "the only intuitive interface ever devised was the nipple"? Turns out there's still a fair bit of active training required all around to even get that going. There's no such thing as intuitive, only familiar.

> Newborns (and certainly toddlers) seem to understand the underlying concepts for these things when it comes to visual/hepatic object identification and "folk physics"

Yes, they enjoy millions of years of pretraining thanks to evolution, ie. their pretrained base model has some natural propensity for visual, auditory, and tactile sensory modalities, and some natural propensity for spatial and causal reasoning.

I'd guess it's because we don't want to have another human. We want something better. Therefore, the expectations on the learning process are way beyond what humans do. I guess some are expecting some magic word (formula) which would be like a seed with unlimited potential.

So like humans after all but faster.

I guess it's just hard to write a book about the way you write that book.

It does but it also generalizes extremely well

I haven't seen a convincing argument that it is more sample efficient than a neural network that has seen an equivalent amount of lifetime information. Yann LeCun gave an interesting explanation of how even in the first few years of a child's life, they have seen much more information than the largest pretrained models have.

Reply on news.ycombinator.com