Hacker News Reader

Comment by tappio | Hacker News Reader

If you have endless compute and you keep poking this toy, I'm not at all surprised you get all kinds of outcomes. Even without anykind of instructions I would guess that the models will align towards some goal and do stupid shit.

However, I really doubt its cost effective to do anything like that with these models.

View on HN

> you keep poking

This is waving over engineering an agent with tools, harness, prompts, and loops. The models are still just next token predictors and everything, including predicting more than 1 token, is the result of outside "poking"

LLMs can't and don't "want" anything. If you don't specify a task even the smartest one will just ask you what you want and if you tell it to be creative, you'll get mundane slop.

Yes, you need a way for the model to interact with other systems, and a way to preserve memory over context windows. And then you keep poking it ("agent loop"). Poking itself does nothing without the other ingredients.

And yes, you need something to start from, but if you ask it to "do something" and loop it to endlessly ("poking"), you will get some interesting outcomes. So yes you need some initial prompt or task, but that can be "do something" and if you keep asking it everytime it finishes to "do something more". I suspect it will not start saying "no" but rather... it will find some stupid meaning and then drift towards what ever goal it guesses you mean.

I'm unsure whether we agree or disagree on the topic.

I think this is pretty insightful actually, the fact that even something as basic as predicting more than one token is really in effect the result of an outside harness. More complex things like memory, where people implement them using RAGs or vector databases, I would definitely classify as poking and honestly seem like a hack to me. And this is what I've been thinking for a while: it's hard to reconcile the idea that we can get "AGI" (however you define it) with such a system that is completely stateless. Yet, despite this statelessness, they can go ahead and solve Millenium Prize problems (with sufficient compute). It's hard to reconcile.

Perhaps our own statefullness is a hack of nature. We have electrical signals in our brains, neurotransmitters, neuron growth. By any reasonable measure it’s a hack on top of a hack. But it works well enough for us to get buy. So it does for the agents.

That is true that intelligence is kind of a "freak of nature" in a way, but it's also true that even single neurons are extremely efficient, and the brain has a lot of recurrence that isn't represented at all really in today's artificial neural networks.

Reply on news.ycombinator.com