Oh for sure. When I present something to the LLM it always tells me how great it is until I make it "question" it, then it says it was overestimating this or that. Eh. Quite annoying.
Half the reason the dominant UX is a "Chat" is that's the only way to provide a facsimile of memory or persistence across requests. Append the last few turns, press go. Over time you can develop an eye for the model's tics/attractor topics.
Remember that they bill by token use, and suddenly, the entire UX/architecture starts making sense.