Hacker News Reader

Comment by pclmulqdq | Hacker News Reader

An LLM being able to pass your graduate-level exam almost certainly means that your exam is bad.

Lots of lazy professors do things like this (true/false and multiple choice answers, very soft questions, etc.), and the presence of GPTs should help them understand that this is not sufficient for evaluating someone's knowledge of a highly technical topic.

View on HN

Scott Aaronson is one of the best professors and researchers in this field in the world.

And the test is actually rather easy. It's an undergraduate test.

It's probably not 'easy' for the average human out there. I would expect >> 90% of humanity to fail that test.

I have highly technical graduate and undergraduate degrees (in CS-adjacent field) and a decade of experience doing software development part time, but no direct experience with quantum computing (or much beyond an undergraduate physics class and an intro CS class in terms of directly related study), and consider myself a better-than-average test-taker. I could get 1D, 1E, 1F and 1T by intuition, 1J by actual knowledge, 1P by tautology, and had roughly no shot at the open-ended questions.

> much beyond an undergraduate physics class and an intro CS class in terms of directly related study

That's puts you in < 1% of the population right there.

That was my point (in agreement with you): I have no shot, and I'm well into the 99th percentile for this test.

Right, forgive me I understood it the other way around.

That doesn't mean anything about the quality of his courses or exams. Lots of world-class professors phone it in on their teaching responsibilities because they don't care. I had several of them in school - usually more prestigious professors had worse courses.

I can guarantee you that ChatGPT cannot do anything that a quantum computing class prepares you to do, aside from passing this final. That makes it a bad test.

ChatGPT, in a sense, is the police on that front.

Oh, you can guarantee, huh? Wow, guess that settles it then.

Reply on news.ycombinator.com