There is nothing special about fatigue or frustration that make it any less modellable than any other implicit structure present in the dataset that it models just fine.
>Though it's super unlikely that it will give the true reason
It's Super Unlikely humans will give the "true reason" for anything they do. There's a fair bit of research that stated reasons for the decisions we make are often(always?) just post-hoc rationalizations even if you believe otherwise.
>very significant gap between them and the idea of an AGI.
What is this idea of AGI that there exists a significant gap still ? It certainly isn't the idea of being Artificial and Generally Intelligent.
>I can confidently say we currently don't know how to imbue agency (as in embodied causal reasoning
I don't see the difference between what you've described and examples like these.