Hacker News Reader

Comment by ilaksh | Hacker News Reader

Could it be named in reference to something like the A* search algorithm? What if Q stands for Query in transformer attention? How would that type of search translate to transformers?

They said that it's able to solve simple math problems. If it's related to A* then maybe it's trying to find a path to something. The answer to a word problem?

View on HN

In reinforcement learning, Q* represents the optimal action-value function

which makes sense. you can pretty easily imagine the problem of "selecting the next token" as a tree of states, with actions transitioning from one to another, just like a game. And you already have naive scores for each of the states (the logits for the tokens).

It's not hard to imagine applying well-known tree searching strategies, like monte-carlo tree search, minimax, etc. Or, in the case of Q*, maybe creating another (smaller) action/value model that guides the progress of the LLM.

Absolutely, maximizing conditional probabilities is easily modeled as a Markov decision process, which is why you can use RL to train Transformers so well (hence RLHF, I've also been experimenting with RL based training for Transformers for other applications - it's promising!). Using a transformer as a model for RL to try to choose tokens to maximize overall likelihood given immediate conditional likelihood estimation is something that I imagine many people experimented with, but I can see it being tricky enough for OpenAI to be the only ones to pull it off.

My bet is a general search algorithm similar AlphaGo that uses LLM as world model, and heuristic search to find the right path to goal. There's already evidence in academics that this can significantly boost model performance [1].

[1] https://llm-mcts.github.io/

My mind went to Q learning.

Ok so maybe nothing to do with A*, but actually a way for GPT-powered models or agents to learn through automated reinforcement learning. Or something.

I wonder if DeepMind is working on something similar also.

If your hunch is right, this could lead to the type of self-improvement that scares people.

Could easily be both

Mine to Q of Star Trek.

But were you thinking of Q, Q, or Q?

My mind went to some kind of Q-learning combined with something like a Monte Carlo Tree Search with some kind of A*-style heuristic to effectively combine Q-learning and with short-horizon planning.

This was alpha-go and alpha-zero right?

likewise. i can already imagine a* being useful for efficiently solving basic algebra and proofs.

it could form the basis of a generalized planning engine and that planning engine could potentially be dangerous given the inherent competitive reasoning behind any minmax style approach.

*(spelled STAR) can also refer to Semantic-parsing Transformer and ASP Reasoner, which while a bit of a stretch, is a concept that shows up in literature.

Reply on news.ycombinator.com