Leela Chess Zero has been around for a while, and it's almost as strong as Stockfish.
It used to be very bad at handicap games because it's trained to maximize the odds of winning and couldn't handle totally lost positions, but they improved it recently. Now it can play at GM level with knight odds.
I don't see why this isn't a focus in RLHF. Just heavily discourage confident lies and encourage admission of ignorance. It can't be that much harder than "safety" training.
But then we would probably end up with things like "As an AI language model, my knowledge of the world is limited, but if I were to guess, ..."