My understanding is that models will often still forget games rules etc unless you design a harness specifically to assist them and prevent them going off the rails. I was going to use chess as an example but it sounds like they are good enough to do that now, albeit I'm not sure if they have a special harness for it.
It's not about how "smart" they are, or how good they are at recall. It's about whether I can teach a model to understand and execute an arbitrary, general task regardless of whether they have been trained on that task before.
Disregarding the above, the fact models still require a human prompter/decision-maker is defacto proof they aren't AGI in my view. If they were humans would be obsolete in decision-making.