Such failures are interesting, in the physicist's sense.

That it struggles with composition sometimes but not other times might indicate model has learned a particular distribution factorization which is good much of the time but makes it unable to properly account for conditional dependencies in certain queries. The issue of losing track of lots of details may also be related.

Its struggles with negation is also fascinating because that has been a thorn since the early days of GOFAI. Negation is trickier and more involved than what one might naively assume.

For those taking relief from such weaknesses, it's worth remembering that this is as bad as it is going to be and probably not for long. Model size is surprisingly small when compared to recent behemoths. That it can already do all that it does is certainly cause for concern.