One of my gotos is a variation of a classic logic puzzle, as models naturally tend to try to provide the original form of the solution because of token similarities and by seeing how well they can identify where steps are wrong or seeing if they can solve the variation when breaking token similarities with representative symbols or other techniques like mapping out object relationships in the initial answer before providing a solution it tends to reveal the underlying capacity for logic outside of simply similarities to the training data.
Mistral, GPT-4, Claude, Gemini can all solve it with various degrees of competency and hand holding.
Llama 3 just cannot handle it.
I think what we're seeing is that the massive training data set is leading to a much more capable stochastic parrot but not necessarily as capable world modeling. Or else there's an advantage to MoE for this kind of task that Llama 3 lacks. But it's tracking with my subjective sense of being somewhere between GPT-3 and 4 in terms of "critical thinking" ability, but not in the same class as more modern models, even though it seems to have a better breadth of training to draw on than most other medium sized models.