I haven't really used LLMs much for coding (sabbatical before LLMs got good at coding, now looking for work) but I found with chats that they are great at exploring well trodden territory but as soon as you go a little bit off the beaten path they flail horribly