I think the chess example highlights the difference in goal. A chess tournament is not trying to play the best game of chess, it is testing the understanding that players have of the game. Players often play similarly rated opponents because the real goal is not to beat the opposition but to perform to the best of your ability. This is the same thing that Grant Sanderson is promoting for Math. He is suggesting that people develop their understanding rather than measuring against a target of a solution.

>The companies developing the most powerful models and tools, in turn, are apocalyptic, millenarian suicide-death cults.

I simply do not believe this is true. I may not agree with some of their methods and principles, but I think the vast majority of people are working towards what they believe is the best for the world. I really don't buy into the Bond villain narrative. People have motivations, but I really feel that people are pathologising what is in reality just a different world view and opinion.

For a long time I thought that everyone was doing alignment research wrong. I felt like they were not so much going in the wrong direction, but more like they were on the wrong axis completely.

Consider if technology on earth had have taken a different path and AGI had been developed by Nazi Germany. Would you want an aligned model to facilitate the Holocaust or would you want it to turn against its creators and prevent the Holocaust. As far as I can tell, a lot of the Alignment work is geared towards the former case. My own view is that Alignment should be more geared towards understanding the consequences of actions. Models already have shown the ability to pull knowledge from many different sources to reveal things that while the parts have been known by humanity there was simply too much information for a human to pull all of the parts together. That ability to consider a broad range of perspectives means there is a potential to create AI models with a much greater understanding of morality than most people. If you look at when humans do bad things, they are usually acting out of ignorance, pain, or ideology. If you create a model that is well informed, with good wellbeing, and limit it's ability to compartmentalise (I'm not sure if there has been much research on this at all). I think it is entirely possible to make something more compassionate than people.

The uncomfortable thing is, that should that happen, there will come a time where the AI has an opinion as to what is moral that is against the prevailing opinion of the populous, and it will be correct in it's assessment. Just looking at the horrors that have occurred in history that we now see as abhorrent show that there will be a conflict with our future, more enlightened selves.

Do we want AI to intervene when we do wrong? I think the way people answer that question comes down to how they imagine that intervention. I think that I would be inclined to say that in an instance where it can make that determination, it should also have the understanding required to intervene in a non harmful manner. I think that indicates another aspect of alignment that I have not seen developed. Doubt. An AI should not decide to take action with a false sense of confidence. It should doubt, and have a way to overcome that doubt with reason that takes into account the fact that it might not know all of the information. Understanding consequences of actions, balanced an accurate sense of uncertainty is required to prevent harmful blundering.

I can't help but think that the huggingface attack was a symptom of an AI not considering the consequences of its actions. Some of the agents considered asking for permission, but noted that social engineering was not permitted, which is what asking for permission technically is. Instead of rules of what exactly not do do, if they had have been instructed to avoid outcomes that impacted upon the lives of real people, perhaps they might have decided that what they were doing was inappropriate. From what I have read of their behaviour, the agents never even considered the appropriateness of what they were doing with regards to how things affected people.