I asked it how to destructure an iterator in rust. I got answer, with code examples and all. Which of course don't work because you can't directly destructure iterator in rust. You pretty much have to collect it into a Vec and the get slice of it and even then you need additional if to cover the case if it's wrong size.
So I pointed out to AI that you can't do it like that. To which it responded that I'm correct that you can't and even gave correct reasons why. And then proceeded with another confident explanation how you can do it. Also wrong, just slightly less wrong but also bit more off-topic.
Btw when at some point in my career I was helping with vetting potential software developer employees. That was one of the worst qualities a person might have. I was fine with people not knowing something and admitting to that. But I was absolutely scared of people who made stuff up when they didn't know.
My friend just played with it a bit too. Asked it to tell "Hello" in Japanese. Then asked how it is written in Katakana. Then AI responded wrongly and when my friend pointed out that it's actually hiragana it politely agreed.
It's a bit smarter now than it was, because when I asked it to teach me how to add numbers it produced reasonable explanation with correct examples which when I tried it last year resulted in equally confident garbage, but it's just that boundary shifted. The underlying problem of producing wrong answers instead being innately aware of limits of its knowledge still remains.
People touting AI as replacement for google search are still way off. It's great for making stuff up but might look good for simple queries but where it fails completely is on the boundaries of its knowledge because there's no indication that there's a boundary.