Comment by simonw | Hacker News Reader

Comment by simonw | Hacker News Reader

> Amazon's hiring algorithm penalized resumes that contained the word "women" in any context. Healthcare risk scoring algorithms used by major US hospitals were found to systematically underestimate the medical needs of Black patients. Apple Card's credit algorithm gave wives credit lines 10x lower than their husbands for the same financial profile.

The Amazon hiring story is from 2018: https://www.reuters.com/article/world/insight-amazon-scraps-...

The "systematically underestimate the medical needs of Black patients" story seems to be this one from 2019: https://www.chicagobooth.edu/research/tolan/research/2019/di...

The Apple Card story is also from 2019: https://abcnews.com/US/york-probing-apple-card-alleged-gende...

None of those stories were about LLMs!

The stochastic parrots paper was published in 2021: https://dl.acm.org/doi/10.1145/3442188.3445922

There's definitely a good, well researched article to be written about the how well the stochastic parrots paper stands up five years later. This is not that article.

I get the sense a lot of the warnings about LLMs were based heavily on known risks of Machine Learning at the time (which those references are all examples of.) That was because the data was relatively narrow (e.g. hiring data.) However the scale of data that LLMs are trained on has qualitatively changed the risk landscape.

Like, before LLMs biases in the data were clearly impacting biases in the model outputs and that was a real risk (e.g. recruiting models deprioritizing minority candidates.) But with LLMs it's not clear that the same risks apply, either due to multiple biases in the overwhelming amounts of data canceling out, or due to RLHF, or some mix of both, or some other emergent property.

The fact that Elon had to deliberately go out and create an "anti-woke" LLM indicates that the models do have biases, but those biases are not the same ones pre-LLM ML safety researchers were concerned about... and may even be aligned with the "well-known liberal bias" that reality has.

I suspect the risks we'll see with LLMs will be very different from what this or older papers focused on.

The scale of the data and the size of the models don't change the underlying issue, the whole construction of these models is to start with a maximum likelihood language sampler (pre-training) and then massage it into a maximum utility language sampler (post-training) with some eye towards risk management and policy compliance ("safety"). It takes work to make model output fit any particular idea of "correct", whether it's Elon's particular ideology, the US Civil Rights act, Xi Jinping Thought, or writing clean C++. More data and weights increase the complexity of tasks that we're able to model but it doesn't automatically make the output "better" on any given axis.

Right, what I meant is the underlying issue is the same, but the large amount of data along with the number of potentially conflicting and reinforcing biases going into LLMs make it hard to categorize or quantify risks.

Like previously it was pretty straightforward to hypothesize and show that "historically minorities were discriminated against in hiring, so models trained on that recruiting data will exhibit the same biases." But now those biases are intermingled with a whole lot of other biases (e.g. including data / RLHF about the ill-effects of discrimination) so it gets harder to reason about their behavior.

As an example, I don't think anyone quite predicted that these could become suicide ideation machines.

Right, but they were about models, and therefore useful to projections about what even more powerful models (LLMs) might do. The author should have made that clear.

The article itself is ai written with purposeful spelling errors. This is classic clickbait content. It’s just not where it normally is; youtube.

Reply on news.ycombinator.com