Hacker News Reader

Comment by simonw | Hacker News Reader

I just published this, as a counter to the "prompt engineering will be made obsolete as AIs get better" argument that I keep seeing: https://simonwillison.net/2023/Feb/21/in-defense-of-prompt-e...

View on HN

We need a name for this, something like genie effect, but that doesn't sound creative enough.

But working of the fables of the genie, we have and all powerful wish machine where the user mistakes they are telling the genie with intention of their thoughts, whereas the genie is looking at all possible decodings of the words they are saying without knowing the users thoughts. If you do not verbalize exactly what you want (and have done some thinking about the ramifications of what you want) you might end up squished under a pile of money, or some other totally ridiculous situation.

Unless you share your entire life with the AI/genie you will always have to give more detail than you think. Hell, in relationships where people live together for years there is always the possibility for terrible misunderstandings when communication breaks down.

The age-old bane of software requirements: "It's exactly what we asked for but not what we want."

I've used an analogy of magic and spell casting to prompting in the past, but I've since been convinced that it's risky to describe it like that because it implies that this stuff is all-powerful and impossible to understand: https://simonwillison.net/2022/Oct/5/spell-casting/

“Any sufficiently advanced technology is indistinguishable from magic” -- A.C.C

I'm wondering if we'll reach a point where it is impossible or nearly impossible, and what value of 'understand' we assign to it. For example deeply understanding x64 processor architecture. Does any one person understand it all, very unlikely at this point. The investment the average person would have to perform to accomplish even part of that would lead most of them to seek other more fruitful endeavors. Biology would be a good example of this. Nothing about biology is magical, and yet even simple systems contain such an information density that they are at the impossible to fully understand point, and that's before we start layering them into systems that have emergent behavior.

It seems to be a theme with AI.

Scale.ai (no affiliation, not a customer) has a product called Spellbook

https://scale.com/spellbook

Yeah, I've definitely had informal conversations with AI experts where they've talked about collecting spells for their spell books!

This was the theme of Bedazzled, the Peter Cook and Dudley Moore film (remade in 2000), itself a twist on the Faust legend.

In this case though, the increasingly desperate engineering of the prompt was each time intentionally side-stepped by the devil, though much like GPT, he couldn't help being like that, it was just his nature.

We need a name for this, something like genie effect

Investigating why the prompt isn't working:

"Debazzling"?

This is great! More experiments are key. I know, I know, many of you are getting flashbacks to physics lab, but come on, it's totally different when you're coming up with the experiments yourself! :D

As for some philosophical inspiration, the analytic philosophers like Kant, Frege, Russel and early Wittgenstein have methods for breaking down natural language into (possibly) useful elements!

Like, everyone speaks of "context" in terms of these prompts... how similar is that to Frege's context principle?

https://en.wikipedia.org/wiki/Context_principle

Some other Wikipedia links:

https://en.wikipedia.org/wiki/Linguistic_turn

https://en.wikipedia.org/wiki/Philosophy_of_language#Meaning

In Frege's The Foundations of Arithmetic he disagrees with Kant that a proposition like "2 + 12 = 14" is synthetic (and Kant used large numbers as an example of how analysis would fail!!!). However, once we have a large language model we can experimentally show that math computations are indeed synthetic. Kant was right!

I actually did read through your "Prompt Alchemy to Engineering" link and the reference to Frege took me very off guard. Then I thought about it a little more and decided mathematical platonism felt even more fitting to me now.

I would love to hear some more of your thoughts, please expand on what you mean!

Reply on news.ycombinator.com