Prompts are Programs | Hacker News Reader

Prompts are Programs

blog.sigplan.org

32 pointsrbanffy28 comments

Command lines are programs. Keystrokes on a calculator are programs. Cooking recipes are programs.

It’s deliberate antagonistic widening of a word’s imprecise meaning, possibly to get some notoriety. Academic clickbait.

Well, I would argue that cooking recipes are algorithms, at least. In fact, they were probably the very first example of an algorithm that was given to you in your first Intro to Algorithms class.

But not programs. Not in the sense of SIGPLAN.

I'm sure I've seen "cooking recipes are programs" back in the late 80s…

Are there any other programming languages that are inherently as imprecise as prompting an LLM? From my knowledge, there isn't, and that seems to be a fundamental diff between prompts and code.

"Take EXPLOR, for example: here is a programming language with probabilistic execution (every instruction specifies the probability with which it should be executed) and no variables -- every instruction operates on either the screen image or other instructions" https://esolangs.org/wiki/Prehistory_of_esoteric_programming...

Thanks for sharing, that was an interesting read.

> every instruction specifies the probability with which it should be executed

Is this still precise, in a way that prompting llms isn't? I'd need to explore the language in more depth to answer that, but if it does what I assume which is basically a string of instructions like do x with probability y, then it's still precise, except we're using probability theory in addition to/instead of standard programming logic.

It has been tried and failed multiple times. I never used one, but IIRC the problem is that you must guess what the language will parse and what it will do.

For example, if you use c in Wolfram Alpha, sometimes it is interpreted as the speed of light, sometimes as centigrade, sometimes as a constant and sometimes as an error. And sometimes minor changes in other part of the question changes the interpretation of c.

von Neumann taught us that code and data are the same thing and the distinction is a matter of context and perspective. Some bits are code-like and some are data-like, and some inhabit a gray area where no bright lines are to be found.

It’s provocative to claim that prompts are more like code than like data. Feels to me more like they sit in that gray area alongside things like HTML, config files, regular expressions, and spreadsheets.

In practical terms the more pressing question is what kind of tools are needed to manage prompt bits. Is it beneficial to manage prompts using traditional software engineering tooling? This post says yes. Maybe, but I can see traditional data management tooling being equally applicable.

Code Is data that expresses instructions to some processing unit for it to do tasks.

Prompts fit that description very well so I don't find that provocative or controversial at all.

The fact that traditionally the execution unit was very rigid and that the language used to express instructions was distant from human natural language is more of a consequence of an implementation rather than a fundamental distinction between code and data

> expresses instructions

To instruct is akin to providing an input. Where do we input to? The program. The program is programmed; but most importantly, the input is not programmed in the sense the program is programmed. It is, perhaps, thoughfully designed (or not). This role is best described as Prompt Designing or Prompt Engineering, not Prompt "Programming". In any case, it is indeed very important. But in my respectful opinion, it is not programming.

It's kind of like thinking that entering a query into a search engine is programming - a host of techniques can be utilized to optimize how that input effects the output (tomes can be written on this just as they can for LLM prompt design by now).

I can kind of defend that premise but it breaks the meaning of conventional usages, and creates more confusion than it brings clarity.

Fwiw the term "prompt engineering" begins as a joke or at the very least half-serious, along the lines of "naming engineer" etc.

But it turns out that a) it's useful to know how to write prompts effectively b) it's time consuming c) it requires skill

So it became more and more an actual thing (discipline?) worth to be taken setiously

All code is data. Not all data is code though surely? Depends how you define "code" I guess. Edit: Thinking about it more, I guess all computer data is "encoded" in some way and represented in a coded form.

This seems to have similar ideas to an post from a year ago by François Chollet: https://fchollet.substack.com/p/how-i-think-about-llm-prompt...

except English or any natural language leaves too much wiggle room and LLMs have .... 0 mappings to machine code. Soon LLMs will do Feature Bloat, Surveillance and Privacy Breaches as a service - no prompt necessary, thank us very much - and then the kids won't even notice who slipped what into their metaphorical drink from that digital fountain. It won't be obvious anymore. Thanks to people saying things like "AI" and "prompts are programs".

> We call the direct input to the chatbot the user prompt. Another, more complex prompt is the prompt that was written to process the user prompt, which is often called the system prompt.

I thought this was called a query, and a prompt was text / symbols written by the system to prompt the user to enter a command or query.

Is this meaning only used by AI folks?

No, one meaning is "an act of encouraging a hesitating speaker," which seems OK to use in this context

I believe the use comes from the concept of a writing prompt, like one might be given for writing an essay or creative writing assignment.

If prompts are programs (which I don't think they are, but they are closer program than search queries), then LLMs don't work as advertised. Which, of course, they don't if you've ever seen a mind-blowing demo and then tried to run some of your own tasks.

Completely agree. We recently published similar thoughts: https://8factoragent.org/

Any interesting examples of methods to make LLMs more "programmable?"

DSPy?

I work at .txt, where we work on exactly this. We use structured generation to make programmable language model systems. Here's two examples I worte + love.

One makes a (sort of) internally consistent infinite world generator with RAG + Outlines:

https://github.com/dottxt-ai/demos/tree/main/lore-generator

This one makes two characters up and then builds a simple combat loop where they fight each other:

https://github.com/dottxt-ai/demos/tree/main/nouscon-24

Both are simple examples of basic programming with language model outputs.

more like undocumented api requests to a remote dev box

Not if you have the weights of the model locally.

No, they’re bullshit. Stop selling bullshit.

No...prompts are definitely not programs, prompts are inputs to programs. How someone can write a blog post on such a demonstrably fallacious premise is beyond me.

I am beginning to write a blog post in rebuttal, because it's so fascinating.

So some text in helloworld.c is no a program because it's passed to another program as an input in order to be executed?

I'd consider it a program, but surely not a deterministic one.

Reply on news.ycombinator.com