andyk

Done! Well, the first suggestion you made anyway :-)

https://github.com/andyk/recursive_llm/blob/main/run_recursi...

    def recursively_prompt_llm(prompt, n=1):
        if prompt.startswith("You are a recursive function"):
            prompt = openai.Completion.create(
                model="text-davinci-003",
                prompt=prompt,
                temperature=0,
                max_tokens=2048,
            )["choices"][0]["text"].strip()
            print(f"response #{n}: {prompt}\n")
            recursively_prompt_llm(prompt, n + 1)

    recursively_prompt_llm(sys.stdin.readline())

> What's the actual goal here?

I tried to expand on my goals and paths I want to explore in a comment below [1], but basically I wonder if we can use this sort of technique as a more powerful version of CoT where prompts can break down a task into sub-tasks (as CoT does) and then recursively do that for each sub-task, until we hit a base-case on all of the sub-sub-...-sub-tasks and (when rolled back up?) the problem is solved.

> You may also be able to get better results by just including the definition of fibonacci in the outer prompt

Yeah, I played with including the mathematical definition of Fibonacci, for example in [2]:

<quote> You are a recursive function ... the paragraph you generate will be an exact copy of this one ... but with updated variables as follows: FIB_INDEX = FIB_INDEX+1; CURR_MINUS_TWO = CURR_MINUS_ONE; CURR_MINUS_ONE = CURR_VALUE; CURR_VAL = CURR_MINUS_TWO + CURR_MINUS_ONE. Otherwise, ... </quote>

[1] https://news.ycombinator.com/item?id=35240093

[2] https://raw.githubusercontent.com/andyk/recursive_llm/main/p...

...and to reply to your second question, one thing I find interesting and want to explore further is how (and when) to best leverage what the LLM has memorized.

The way humans do math in our heads is an interesting analog: our brain (mind?) uses two types of rules that we have memorized:

1. algebraic rules for rewriting (part of) the math problem

2. atomic rules things like 2+2=4

So I'm wondering if we could write a "recursive" LLM prompt that achieves a similar thing.

Related to this, as part of another classic CMU AI research project on Cognitive Architectures, John R. Anderson's group explored how humans do math in their head as part of his ACT-R project: https://www.amazon.com/Soar-Cognitive-Architecture-MIT-Press...

The ACT-R group partnered up with cognitive scientists & neuroscientists and performed FMRIs on students while they were doing math problems.

Yeah, I see the similarities! I like the idea of the prompt containing context that the resulting prompt is going to be executed at the terminal.

My personal point was just to document my exploration of using prompts to generate new prompts, and more specifically the case where the prompts contain state and each recursively generated prompt updates that state to be closer to an end goal (which in this case I compared to the concept of a base case in recursion).

For whatever reason Patrick H. Winston's MIT OCW lecture on Cognitive Architectures always stuck with me, and in particular his summary of the historical system from CMU called General Problem Solver (GPS) in which they try to identify a goal and then have the AI evaluate the difference between the current state and the goal and try take steps to bridge the gap.

https://www.youtube.com/watch?v=PimSbFGrwXM&t=189s

The ability for LLMs to break down problem into sub-steps (a la "Let's think step by step" [1]) reminded me of this part of Winston's lecture. And so I wanted to try making a prompt that (1) contains state and (2) can be used to generate another prompt which has updated state.

[1] Large Language Models are Zero-Shot Reasoners - https://arxiv.org/abs/2205.11916

It is quite unstable and frequently generates incorrect results. E.g., with the Fibonacci sequence prompt, sometimes it skips a number entirely, sometimes it produces a number that is off-by-one but then gets the following number(s) correct.

I wonder how much of this is because the model has memorized the Fibonacci sequence. It is possible to have it just return the sequence in a single call, but that isn't really the point here. Instead this is more an exploration of how to agent-ify the model in the spirit of [1][2] via prompts that generate other prompts.

This reminds me a bit of how a CPU works, i.e., as a dumb loop that fetches and executes the next instruction, whatever it may be. Well in this case our "agent" is just a dumb python loop that fetches the next prompt (which is generated by the current prompt) whatever it may be... until it arrives at a prompt that doesn't lead to another prompt.

[1] A simple Python implementation of the ReAct pattern for LLMs. Simon Willison. https://til.simonwillison.net/llms/python-react-pattern [2] ReAct: Synergizing Reasoning and Acting in Language Models. Shunyu Yao et al. https://react-lm.github.io/

Good question. We are betting that the cost per LLM inference will continue to go down for given level of performance as the market matures over the next few years.

Yeah, exactly "no code first", which I hadn't heard of before. Cool.

hal9 is cool, and in a similar spirit. One thing we've observed with many approaches being explored lately is that they generate code into an IDE like experience. We are more of the camp of making the code gen and code run happen under the hood, so that you get an end-to-end experience ("ask question" -> "see result of running generated code").

Then still allow the user to inspect the code generated, fix it if necessary, and re-run. It's a bit of a subtle distinction but we think that UI/UX makes a big difference in the utility of these LLM applications.

Good ideas.

Business model wise, beyond this interface we have a demo of natural language to data-code (SQL and Python pandas/matplotlib/etc. APIs).

Cool project. I feel like the punchline got buried a little in the article.

"It is being used by more than 10 organizations for a diverse set of use cases, including: model training on GPU/TPU (3x cost savings), distributed hyperparameter tuning, and bioinformatics batch jobs on 100s of CPU spot instances (6.5x cost savings on a recurring basis)."

I don't see a list of the 10 orgs using it, though I do see they discuss at least one: the Salk Institute for Biological Studies.

I wonder if the team has a "powered by" list somewhere (which is pretty typical for OSS projects)?

Awesome.