They’re mediocre to awful at consistently following instructions, unless you have the fortune of having a task and domain that are well represented in the post-training data.
Yesterday I needed to generate ~500 filenames given a (human written) summary of each document’s contents. This seemed to be the perfect task to throw an LLM into a for loop. Yet it took three hours of prompt engineering to get a passable set of results - yes, I could’ve done it by hand in that time. Each iteration on the prompt revealed new blind spots, new ways in which the models would latch onto irrelevant details or gloss over the main point.
That would definitely account for the difference in my perception of their utility vs other peoples’.
Applied to ChatGPT's 4o:
> "Suggest good guidelines for generating ~500 filenames given a (human written) summary of each document’s contents"
> "Pretend you have this data and create 10 example file names."
> "Financial_Report_Q1_2024.xlsx (Summary: "Quarterly financial performance for Q1 2024, including revenue and expenses.")"
> "Marketing_Strategy_SocialMedia_2024.docx (Summary: "Comprehensive marketing strategy focusing on social media growth for 2024.")"
> "HR_Employee_Handbook_v2.1.pdf (Summary: "Updated version of the company employee handbook with revised policies.")"
And so on. In general, you have to either create the process or have it create a process and refine it if it isn't quite there. You can even use standard processes and guidelines when they're available. Then it usually does a great job.
The reasoning models improve on this kind of thing, but that just means they do even better when provided with or asked to create and follow a process. o3-mini-high, since it shows its work, didn't just spit out a process. It showed the process of creating the process in its reasoning block. It considered the format above in there and removed and added things while considering different levels of detail before the final version before producing anything.