Mass attempted replication of old findings of LLMs with new models

Have LLMs fundamentally actually changed or improved to resolve problems like this, or have the big models been aggressively tuned / trained on specific examples so it’s less embarrassing?

Because I’ve noticed that Google’s stuff always seems to generate code to do counting tasks, even when not asked for, so I wondered if the LLM itself even generated the answer or it’s just being instructed to use a tool? (Which IMO is a workaround)