It would be interesting to see if it can also play chess with altered rules, or actually just a novel 'game' that relies on logic & reasoning. Still not sure if that would 'prove' LLMs do reasoning, but I'd be pretty close to convinced.
I wonder if there are variants that have good baselines. It might be tough to evaluate vis a vis human performance on novel games..