Hacker News Reader

Comment by KhoomeiK | Hacker News Reader

A group of PhD students at Stanford recently wanted to take AI/ML research ideas generated by LLMs like this and have teams of engineers execute on them at a hackathon. We were getting things prepared at AGI House SF to host the hackathon with them when we learned that the study did not pass ethical review.

I think automating science is an important research direction nonetheless.

View on HN

That’s pretty wild. What was the reason behind failing ethics review?

I'm generally a proponent of AI and LLM but to me the decision was the right one. You are tasking people with implementing an idea generated by an algorithmic model with (I'm guessing) zero oversight that might have very little training that teaches it the importance of coming up with ideas worth implementing. Some may be more useful than others so it won't be fair from an accomplishment or motivation point of view.

Imagine you've already invested time going to this event and want to win the prize/credit but to do so you have to implement a plugin that makes webpages grayscale because of a random idea generator. Maybe some people would find that interesting but others would see it as wasting their time.

As long as all participants are well-informed then there is absolutely no ethical issue...

How do you make sure the participants are well informed? What if an idea suggested by a model turns out to be dangerous to implement, but nobody at the hackathon has quite the relevant experience to notice?

Such as?

rsfern is asking exactly that

No, I'm asking for an example of an idea that an LLM might produce that is too dangerous to implement but nobody at the hackathon has the relevant experience to notice. You can shut down any endeavour by imagining boogeymen that aren't actually real.

I don’t think anything needs to be shut down necessarily, I’m just suggesting reasons why a reasonable ethics board might be hesitant to green light such a hackathon if it’s not clear the organizers have done their due diligence on safety

I might be biased in terms of the safety profile, my background is materials and chemistry, and there are loads of ways you can get into trouble if you don’t really have experience in the materials and synthesis routes you’re working with

One example I’ve heard of from my field (alloy design) is an ML model that suggested an composition high in Magnesium - perfectly reasonable if you’re interested in lightweight strong alloys, but the synthesis method was arc melting, which is a high risk for starting a metal fire if you aren’t careful because Mg has a low vapor pressure

If you’re doing organic chemistry it’s maybe even worse because there can be all kinds of side products or runaway exothermic reactions, and if you’re doing novel chemistry it might take deep experience in the field to know of those things are likely

All these concerns are manageable, but I think an ethics review panel would want to at least see that there is a reasonable safety review process in place before letting students try out random experiments in topic areas in which the models likely haven’t been fine tuned with safety in mind.

Surely the ideas themselves are what should be examined for ethical suitability, rather than the meta-idea of “ask an LLM for ideas”?

Individual ideas can be subject to Ethical Review Board approvals and that should go for a hackathon project same as any study proposed in Academia or drug trial etc -- but to apply some wavey handed lum sum out of bounds lable just based on source seems like arbitrary opinionated overreach.

One obvious problem is, what if the ideas were obviously unethical?

I would personally let this pass ethics if someone read all the generated ideas, and took personal responsibility for them passing the basic ethics rules, or got them through the ethics committee if required, exactly the same as they would their own ideas.

I don't think LLMs are the right approach for this. Coordinated science would basically be a search problem where we verify different facts using experiments and use what we learn to determine what experiment to do next.

When you can run experiments quickly it becomes feasible to use ML and evolutionary methods to do novel discoveries, like AlphaTensor's better matrix multiplication than Strassen, and AlphaZero's move 37, upturning centuries of game strategy.

The paper "Evolution through Large Models" shows the way. Just use LLMs as genetic mutation operators. Evolutionary methods are great at search, LLMs are great at intuition but get stuck on their own, they combine well. https://arxiv.org/abs/2206.08896

The interplay between LLMs and Evolutionary Algorithms, despite differing in objectives and methodologies, share a common pursuit of applicability in complex problems. Meanwhile, EA can provide an optimization framework for LLM's further enhancement under black box settings, empowering LLM with flexible global search capacities.

Since chatGPT was first released hundreds of millions of people have been using it for assistance, and the model outputs influenced their actions, maybe even supported scientists to make new discoveries. The LLM text is filtered through people and ends up as real world consequences and discoveries that are reported in text, and get in the next training set closing the loop.

Trillions of AI tokens per month do this slow feedback game. AI speeds up the circulation of useful information and ideas in human society, and AI feedback gets filtered by the contact with people and the real world.

Reply on news.ycombinator.com