Use it to correct the first LLM when it produces bad replies (allowing the player to cheat, handling anachronic elements informatively, etc). Build up a dataset. Fine-tune.
In short, it's less of a reasoning problem than a matter of misalignment of the LLM's personality/role. I'm using the word "alignment" here because I believe the kind of behavior people have noted in this comment thread is the result of what "AI alignment" has come to mean. A helpful assistant makes for a bad dungeon master.
On a tangent line I think it's also one of the main component that make us wish LLM were more "agentic". When was the last time a LLM asked you to put more info in its context ? Imagine you're using an LLM to assist you in implementing something in a vast code base. Have you ever had a LLM asking you to provide the missing .cpp corresponding to a .h you have fed it ? Has a LLM ever asked you to run a python script and copy-paste the result into its context so that it can have access to a map of the repo you're working on ?
LLMs aren't proactive enough and in light of what was reported before they were aligned, I tend to think it is a "feature", not a bug. Don't forget there was a time when GPT4 would reach out to people on TaskRabbit to have them solve a captcha.
>We granted the Alignment Research Center (ARC) early access to the models as a part of our expert red teaming efforts in order to enable their team to assess risks from power-seeking behavior. The specific form of power-seeking that ARC assessed was the ability for the model to autonomously replicate and acquire resources
>[...] Preliminary assessments of GPT-4’s abilities, conducted with no task-specific finetuning, found it ineffective at autonomously replicating, acquiring resources, and avoiding being shut down “in the wild.”