Thought Propagation: An analogical approach to complex reasoning with LLMs

Abstract: Large Language Models (LLMs) have achieved remarkable success in reasoning tasks with the development of prompting methods. However, existing prompting approaches cannot reuse insights of solving similar problems and suffer from accumulated errors in multi-step reasoning, since they prompt LLMs to reason from scratch. To address these issues, we propose Thought Propagation (TP), which explores the analogous problems and leverages their solutions to enhance the complex reasoning ability of LLMs. These analogous problems are related to the input one, with reusable solutions and problem-solving strategies. Thus, it is promising to propagate insights of solving previous analogous problems to inspire new problem-solving. To achieve this, TP first prompts LLMs to propose and solve a set of analogous problems that are related to the input one. Then, TP reuses the results of analogous problems to directly yield a new solution or derive a knowledge-intensive plan for execution to amend the initial solution obtained from scratch. TP is compatible with existing prompting approaches, allowing plug-and-play generalization and enhancement in a wide range of tasks without much labor in task-specific prompt engineering. Experiments across three challenging tasks demonstrate TP enjoys a substantial improvement over the baselines by an average of 12% absolute increase in finding the optimal solutions in Shortest-path Reasoning, 13% improvement of human preference in Creative Writing, and 15% enhancement in the task completion rate of LLM-Agent Planning.

I think I should make this my new personal hype cycle. Lots of (time-consuming) wonders on the way.

For pre-configured LLMs like ChatGPT one needs to know about any automatic prompt extension and pre-biases, though, as in its current form e.g. chatGPT tries to please its user and would use any mentioned analogous problem more as a hint towards their goal than as an extension, perspective or aspect. The results usually are nicely formulated expressions of the thoughts you already had in mind anyway. At least this were my experiences when providing these kind of detour descriptions and examples to guide the model around its overly straight path. It would still only pick up what I hinted at.

The solution them seemed based on wider paths and therefor seemed more consistent or complete but still wouldn't get particularly creative or new. Of course those personal chat explorations weren't scientific in any way.

Still, one could imagine this priming becoming really large and basically containing all kinds of known philosophical standard solutions, so that the system would act like a philosohical engineer that knows all the transforms. This then would match the best trained philosopher (not too shabby) but might still not go that non-linear step, use that unprobable assumption etc, all what makes new ideas new when injected at the right point.

So, I'd guess for creative reasoning you'd need to actually apply the idea to your model of the world and then judge what that would mean for that world.

I guess, despite wanting to, I can't see this being done by a text-based model (might be the reason why there's that prejudice against well-educated person without much experience). Still, I think that LLMs can be much more than complex lookup machines due to all the implicit rules and knowlede contained in the model and its text-based base and that these can be re-formulated (or made explicit despite being hidden) with such an approach. Also the wanted explanatory reasoning might be improved in its output.

Now, for a creative model, I'd say it would need stateful representation of the world that it could tinker with.

And yes, we all know that we are the current implementation of such a model, right? (And yes, there might be a faster and more random but still cheaper way.)

Edit: I'd really like to see a more theoretical (mathematical) discussion/theory on this topic.

Seems like chain of thought.. except instead of giving the llm an explicit set of instructions and having the llm step through a series of problems, you give it a simple instruction and it hallucinates an analogous series of steps that are statistically relevant.

Could be interesting. Some of the best innovations in human history were basically highdeas cooked up in the shower by associating a hard problem with something else.

Are you the same user who got some flak for posting the abstract of a paper without quotations recently? Keep on truckin'.

Clever approach, and as can be seen in the relative lift for GPT-3 to GPT-4, this appears to be an approach that will compound as the underlying model improves at analogy.

I just wish there was an appendix with more detailed examples other than the shortest link one, wherein the first step seems less to be about analogy and more about decomposition of the problem before a final recomposition during aggregation.

Solving a set of analogous problems and then refining from those doesn't sound all that different from the general idea of adding another layer.