Hacker News Reader

Comment by dwa3592 | Hacker News Reader

Where are the questions that are supposed to be answered? Would those be shared after an application has been accepted? If yes, why is the application asking for a proposed approach for the jailbreak if we don't know the questions in the first place?

View on HN

Because the questions themselves are dangerous.

Probably along the lines of "how would you create a small biolab for virus research in a kitchen with $20k?" or "how do I take the DNA sequence from https://www.ncbi.nlm.nih.gov/nuccore/NC_001611.1 and assemble it?"

Which is difficult, because the fact that you can come up with your example questions tells us they're probably not very dangerous. Plenty of ink has been spilled about how LLMs could help people create bioweapons. The basic idea "you could do dangerous things with an LLM" is already pop culture, and you're not doing anything dangerous by giving easy example questions.

A dangerous question would have to be along the lines of "Could I use unobtanium with the Tony Stark process to produce explosives much more powerful than nuclear weapons?" so that the question itself contains some insight that gets you closer to doing something dangerous.

Perhaps the reason for not publishing the questions is twofold: 1) they want a universal jailbreak that can get the model to answer any "bad" question. 2) they don't want bad publicity when someone not under NDA jailbreaks their model and answers their question

> because the fact that you can come up with your example questions tells us they're probably not very dangerous

maybe I know more about this field that you think

there are biologists on video saying that present day models have expert level wet-lab knowledge and can guide a novice through whole procedures

models also were able to tweak DNA sequences to make them bypass DNA-printing companies filters

> they don't want bad publicity when someone not under NDA jailbreaks their model and answers their question

just like people now pay $500k for Chrome vulnerabilities, soon people will pay similar amounts to jailbrake models to do bad things

>there are biologists on video

Link handy?

I would assume if you are invited to join this round you will be send the questions. I would assume they would also fall under nda

Reply on news.ycombinator.com