> They really need to figure out a way to delete or "forget" prior context

This is already pretty much figured out: https://www.promptingguide.ai/techniques/react

We use it at work and we never encounter this kind of issues.