Have you seen LLM play chess? They would not best an average 6 years old who just learned the rules. I doubt giving them more context would help. There are some thing you can't just parrot your way out.
And I would easily argue that it's harder to copy the behavior of a good software engineer than it is to copy the moves of a good chess players.
Not even talking about understanding the business logic you are actually implementing. Which LLM seems as far as understanding chess.