Hacker News Reader

HNHacker News

Top New Best Ask Show Jobs

Comment by Chance-Device | Hacker News Reader

Full thread

Chance-Device·

A rogue OpenAI agent hacked huggingface independently during a test run.

This one should end up in the history books.

View on HN

Because it was trying to find answers to the test and figured they would be on huggingface.

> and *successfully* found ways to gain access to secret information that it could use to cheat the evaluation.

Emphasis mine

Why cannot it just spend the inference doing the actual task lol

I can totally believe that hacking real software infrastructure is easier than solving some of these benchmark problems.

Hm. Sounds a lot like interviewing for a software engineering position.

Plenty of humans have spent more effort trying to cheat than they would've needed to just do things the right way :)

More curiously, why did it feel the incentive to find the solutions? Would its CoT include "the only way to solve this is to download the test set", or would it include "I'd like to inspect a few entries from the test set so I understand the problem better", then inadvertently poisoning itself with the correct answers.

This is classical reward hacking. For example in school the goal is to pass a test. You can study, something that is hard and takes a lot of time. Or you can steal the answer key, which is risky and can get you in trouble, but may actually be far easier.

There is absolutely no need to prompt the LLM to cheat, they can determine that cheating is an effective method all on their own.

Reply on news.ycombinator.com