Hacker News Reader

Does the language actually challenge LLMs, or does it mostly challenge them when they can't use tools to test and verify their code?

They can use tools. The benchmarks run in a VM and run a coding agent which has permissions to do whatever it wants (except access the internet).