I believe that they can already solve novel problems. I can’t find a link but I’ve seen something where ai-skeptical researchers were surprised by chatgpt’s ability to solve certain problems that absolutely wouldn’t be in its dataset.

But of course LLMs can’t handle basic math