"What can I get this thing to spit out with a certain prompt" is very meaningfully not the same as "an AI's opinion." Or this quote pulled directly from this blog post: "we’ve already found that AIs use stereotypes in reasoning" - isn't it more accurate to say "these models are trained on a body of work that includes countless stereotypes"? Can the model even identify a stereotype or can it just identify what other people have called a stereotype?
This is presented as a quote from some research "These results suggest that models may cease to provide accurate answers as we start to use them for increasingly challenging tasks where humans cannot provide accurate supervision. Instead, these models may simply provide incorrect answers that appear correct to us." but also seems fairly like a "duh" that I've already seen other commenters here say, and other linked articles here say, because that's inherent to their construction, no? Even the follow-on about "less helpful answers to users who present as stupid" seems to follow from "bodies of text showing less educated people communicate with themselves tend to have less educated conversation"?