Comment by TobyTheDog123 | Hacker News Reader

Comment by TobyTheDog123 | Hacker News Reader

I have one of these jobs currently.

It is a nightmare for me, and I do somewhat regret taking the contract even with the sizable hourly rate.

In my eyes, programming makes sense. Even if I introduce bugs, I can sooner or later track down the issue, facepalm, resolve it, and move on with my day, with some semblance of accomplishment and lessons learned.

My prompt engineering work offers no such rewards, and is a total time-suck. This is because, as another commenter wrote here, it is "throwing shit at the wall and hoping it sticks."

While you can treat it as a scientific endeavor, testing hypotheses against a black box, you will never find a prompt that works consistently, even with a low temperature, solely because these models were not built to give consistent results. There is no end-all solution, there is no "correct prompt."

Companies employing prompt engineers are looking for such consistency. Prompt engineers are therefore stuck in a system where they simply cannot succeed. They can hope and pray that the testing done by managers produces fruitful results upon every test, but the results are, for all intents and purposes, random.

Thank you for the honest peek behind the curtains.

Are the companies enamored with the productivity gains of prompt-based systems sufficiently that they can ease up on the requirement for consistency?

In most cases prompt engineers are hired solely to achieve consistency, in order to reliably integrate the responses into other areas of their stacks which may only be able to handle certain kinds of inputs.

Let me give a real example (cant go into too much depth but this is generic enough). I was told to create a prompt that would make ChatGPT generate ten of the hottest real estate markets:

"You must always format this data in exactly the following format: ${index (as a single numeral)}: ${name of location} ${median housing price} ${YoY increase} ${link to internal system} ${number of available locations}"

You would think this would be a pretty easy task for ChatGPT/Davinci to handle, but like I said, consistency is completely missing.

- Sometimes it will say "Market ${index}" instead of just the number, even with the single numeral qualifier.

- Sometimes it will comma separate the data, even if there are explicit instructions not to do so

- Sometimes it will provide the prices/percentages/numbers as regular integers, sometimes it will format them, again, even if explicit instructions are provided

- Sometimes it will place an empty space in between the lines, even if there are explicit instructions not to do so

- Sometimes it will place random punctuation at the end, even if there are explicit instructions not to do so

It's entirely random, you could have a 4000 token prompt or a 40 token prompt, and the results will be of the same quality.

Usually, this is fine for content generation. But content formatting? Anything remotely specific? Forget about it. All of the models (I haven't been able to try ChatGPT 4 yet, however) will simply ignore things, even if (and I promise it's the last time I'll say this) there are explicit instructions not to do so.

Does it help if you give it a few examples of correctly formatted data instead of trying to describe the desired format to it?

Unfortunately this suffers from the same effect as the descriptions. I would say a solid 7 times out of 10 everything works as expected, but sometimes it seems the model goes down another path and ignores most instructions.

I would claim that using examples do improve its chances, but it is still too random to assume the edits were what improved the response, or whether it just happened to be a lucky roll of the dice.

Very not my field, but I've seen it suggested that attempting to highly constrain response format can degrade semantic quality, so one does separate report and reformatting prompts?

Perhaps ask for json, to simplify the output space?

I imagine in the near future we'll be able to prompt with a json schema, and have it enforced.

Reply on news.ycombinator.com