Thank you for the honest peek behind the curtains.
Are the companies enamored with the productivity gains of prompt-based systems sufficiently that they can ease up on the requirement for consistency?
Are the companies enamored with the productivity gains of prompt-based systems sufficiently that they can ease up on the requirement for consistency?
Let me give a real example (cant go into too much depth but this is generic enough). I was told to create a prompt that would make ChatGPT generate ten of the hottest real estate markets:
"You must always format this data in exactly the following format: ${index (as a single numeral)}: ${name of location} ${median housing price} ${YoY increase} ${link to internal system} ${number of available locations}"
You would think this would be a pretty easy task for ChatGPT/Davinci to handle, but like I said, consistency is completely missing.
- Sometimes it will say "Market ${index}" instead of just the number, even with the single numeral qualifier.
- Sometimes it will comma separate the data, even if there are explicit instructions not to do so
- Sometimes it will provide the prices/percentages/numbers as regular integers, sometimes it will format them, again, even if explicit instructions are provided
- Sometimes it will place an empty space in between the lines, even if there are explicit instructions not to do so
- Sometimes it will place random punctuation at the end, even if there are explicit instructions not to do so
It's entirely random, you could have a 4000 token prompt or a 40 token prompt, and the results will be of the same quality.
Usually, this is fine for content generation. But content formatting? Anything remotely specific? Forget about it. All of the models (I haven't been able to try ChatGPT 4 yet, however) will simply ignore things, even if (and I promise it's the last time I'll say this) there are explicit instructions not to do so.
I would claim that using examples do improve its chances, but it is still too random to assume the edits were what improved the response, or whether it just happened to be a lucky roll of the dice.
Perhaps ask for json, to simplify the output space?
I imagine in the near future we'll be able to prompt with a json schema, and have it enforced.