Hacker News Reader

Comment by bearjaws | Hacker News Reader

I had to "AI" proof my technical screening questions for engineers. I was alarmed that about 1/3rd of my questions were easily answered by ChatGPT. Thankfully I've always made about 1/3 my questions from their resume so hopefully they can answer that without GPT...

Most of the ones that it did not answer you could tweak the question into ChatGPT to get a correct answer, but would lead to a pretty noticeable delay in answering.

View on HN

After reading your reply here, I did the same with a question that I've calibrated over 100s of in-person interviews with college candidates at Microsoft. This is for both intern and full-time candidates. The goal of the question was to probe for technical competence (my role in the loop). The assumption was that anyone who can remember their data structures and algorithms class can make it through maybe 50% of the question. The question got progressively more challenging so that I could see what happens when the candidate reaches the edges of their knowledge.

It starts very fizz-buzzy and if the candidate makes it to the end, there's a deeper discussion of caching impacts on performance and optimizing algorithms. ChatGPT nailed it. Even when I said things like:

"can you optimize this program further to maximize the utilization of L2 caches in modern CPUs?"

And it did it in <10 minutes. The best candidate I ever saw took 25 minutes and the rest of the candidates took the full allotment of time 45 minutes and none of them got to the discussion of L2 cache optimization. These are candidates from the best schools in the country.

This was really impressive.

The bottom line being, you don't need to hire engineers anymore, just ChatGPT operators?

We are fucked way sooner than we anticipate. Either transformers level out somewhere right around here to give humans a few years (really a decade would be nice) to prepare, or we are going to slam into an event horizon that's impossible to see the other side of. It's unknowable how humanity will react to be pounced on by commoditized intelligence.

ChatGPT feels like something that is ahead of schedule. Years ahead of schedule.

> And it did it in <10 minutes. The best candidate I ever saw took 25 minutes and the rest of the candidates took the full allotment of time 45 minutes and none of them got to the discussion of L2 cache optimization. These are candidates from the best schools in the country.

I'm not surprised. It's something that's covered very well by introductory textbooks and college level labs. Both of which are great training material for GPT-3. There's only so much complexity that can be crammed into a ~1 hour interview problem.

Ha, interesting. I just tried asking my go-to interview question. It started out writing a correct but way too long solution, so I asked if it could make the solution shorter, and it figured out the concise way to do it. Which is exactly what happened 80% of the time when I asked the same question in dozens of real interviews. It thinks so much like a human, it's uncanny.

I'm not sure if you've ever interviewed software interns, some just first or second year undergrads. Making those questions AI proof is significantly harder. The coding questions must be very easy. I wrote about that here, with examples:

https://blog.stuartspence.ca/2022-12-chatgpt-eccc.html

Reply on news.ycombinator.com