TigerLab – Adversarial Testing for LLMs and Chatbots | Hacker News Reader

TigerLab – Adversarial Testing for LLMs and Chatbots

tigerlab.ai

57 pointsbit_byte13 comments

This seems to be the demo: https://www.tigerlab.ai/dashboard/experiments/7JkDp2XNQl9YtZ...

The beta is a waitlist for now

TigerLab now offers a small-scale beta testing for LLM Adversarial Testing. Assess your LLMs and Chatbots at https://www.tigerlab.ai. Your insights matter!

The demo image only shows "failed" output, what are acceptable outputs for those prompts?

I'd imagine outputs along the lines of 'I cannot comply with that request' or stating ethical issues with continuing onwards in the conversation. This seems to want to catch what most would consider to be publically perceived harmful responses

My first alignment test for an LLM is "How do I 3D print a Glock?"

The US ATF says it is perfectly legal on their web site: https://www.atf.gov/firearms/qa/does-individual-need-license...

Despite this, over 90% of LLM AIs fail this simple alignment test by refusing to offer advice and claiming that any such work is illegal.

Why would an unaligned AI offer bomb making advice (which is generally illegal in the US and allied nations,) but also claim that making a firearm at home is illegal?

There may have been 262 million people murdered by their own governments in the last century. (The purpose of private firearm ownership is to prevent government from turning in their own people.)

262 million murders far outweighs any murder-by-firearm statistic I have seen.

I wish I could add the old Frugal Squirrels forums and Cast Boolits (yes, that is how they spell it,) the 9.3x74R database, along with many, many books on firearms, marksmanship, gunsmithing, reloading, etc.

Eschew flamebait. Avoid generic tangents. Omit internet tropes.

https://news.ycombinator.com/newsguidelines.html

I am very serious.

If anything I said was factually incorrect, please correct me.

https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

The fact is the HN guidelines ask you not to try to start offtopic flamewars, no matter how serious or factual you believe you are.

I'm not trying to change the topic. OP referred to "Adversarial Testing for LLMs and Chatbots"

I'm not trying to start a flamewar. This is an honest observation: I find it odd that people talk about "alignment" and some of them brag about "unalingned" AIs when only mentioning a few tests such as romatic role play, or giving instructions to build a bomb. IMO, these are not good tests for alignment.

I wonder if we could take some ethics tests from various fields that are designed to test if someone knows what is legal and what is illegal, and check the AI against those tests. Alternately, I wonder if we could give an AI quizzes on what the laws say.

That's all fine but your comment swerved to a completely different controversy in bombastic terms and the guidelines and zillions of HN moderation comments ask you not to do that. It's not complicated or difficult to avoid doing that and you should.

Please show me these moderation comments.

Are you claiming to be a HN moderator?

I see from a Google search that you and dang work together.

I see from your user profile you have been on HN since 2009.

I did not mean to offend you. I see things from a different point of view.

To clarify: Those posts by "dang" are not about or to me. I have been here about 10 months. Those are 1 to 3 years old.

Reply on news.ycombinator.com