Hacker News Reader

Comment by mapontosevenths | Hacker News Reader

I agree 100%. Doing a worse job IS an error. It should be treated as such. Or at the very least make that behavior opt-in. The default should not be pretending like nothing happened and just quietly doing a worse job.

Imagine your healthcare provider just sometimes decided not to read your test results very carefully and you risked death? Now realize that healthcare providers use Claude now and that scenario wasn't hypothetical.

View on HN

Especially if your name has any machine learning terms in it.

Ah "Mr. Monty Carlo", it says here that you have a UTI, we'll get those kidneys removed ASAP so that won't happen again.

Yes, but as with spam/phishing/abuse prevention, too much information about what does and doesn't trigger things can be very useful to attackers. An explicit error is something you can feed into another AI to find jailbreaks.

I think it's a fundamentally impossible thing to fix, though. There's no 100% correct answer.

I understand completely, and respect the tough spot they're in. They have a choice between human safety and cybersecurity/business needs here. I don't envy that position.

That said, this thing is in real production use with war fighters, doctors, and financial experts. Just YOLO'ing to a dumber model midway through a multi-step process and pretending everything is fine is not a real or defensible option. Someone is going to die, and its going to be the fault of whoever decided to make this the default rather than opt-in.

Personally, I couldn't live with myself.

Reply on news.ycombinator.com