After reading your reply here, I did the same with a question that I've calibrated over 100s of in-person interviews with college candidates at Microsoft. This is for both intern and full-time candidates. The goal of the question was to probe for technical competence (my role in the loop). The assumption was that anyone who can remember their data structures and algorithms class can make it through maybe 50% of the question. The question got progressively more challenging so that I could see what happens when the candidate reaches the edges of their knowledge.

It starts very fizz-buzzy and if the candidate makes it to the end, there's a deeper discussion of caching impacts on performance and optimizing algorithms. ChatGPT nailed it. Even when I said things like:

"can you optimize this program further to maximize the utilization of L2 caches in modern CPUs?"

And it did it in <10 minutes. The best candidate I ever saw took 25 minutes and the rest of the candidates took the full allotment of time 45 minutes and none of them got to the discussion of L2 cache optimization. These are candidates from the best schools in the country.

This was really impressive.