Sure, that wouldn't be impressive if you wrote tree search for chess. However, if you wrote a program that doesn't do any search and still plays well, that'd be impressive. I don't understand your claim that LLMs playing chess is anything sort of understanding.
related: https://parrotchess.com/