I’ve never heard this term “AI alignment” before and they also don’t define it…
The point is "aligning the AI's goals with human interests", i.e. make the AI non-evil.
You're somewhat lucky to never have gone down that particular rabbit hole. Some strange, but also seemingly very intelligent people on the internet argue that it's the most significant extinction risk humanity faces. My impression is that actual AI/ML researchers believe this to be mostly bullshit. (I'm personally not qualified to decide, just explaining the positions.)
The author of this curriculum seems to define it as:
> Within the coming decades, artificial general intelligence (AGI) may surpass human capabilities at a wide range of important tasks. We outline a case for expecting that, without substantial effort to prevent it, AGIs could learn to pursue goals which are undesirable (i.e. misaligned) from a human perspective. We argue that if AGIs are trained in ways similar to today’s most capable models, they could learn to act deceptively to receive higher reward, learn internally-represented goals which generalize beyond their training distributions, and pursue those goals using power-seeking strategies. We outline how the deployment of misaligned AGIs might irreversibly undermine human control over the world, and briefly review research directions aimed at preventing this outcome.
This is from here, which I haven't read, but found three links deep on the posted site: https://arxiv.org/abs/2209.00626
And no peer-reviewed research, apart from the dozens of widely-cited academic papers on this? All of these are just from one lab, not even including groups like CHAI: https://www.anthropic.com/#papers
Which of these papers track AGI Safety and are also published in peer reviewed journals like SIGKDD, NeurIPS, AAAI etc?
Open AI published a number of papers, essentially and implicitly supporting the position of the AI inexperienced founders. They try to highlight AI safety aspects quite frequently. But I would not take it too seriously, just like you wouldn't take a Facebook employees endorsement of Zucks vision.
If you are familiar with ML, you would know that L2 norms are a key optimization metric in ML. L2 norms are basically Pythagorean hypotenuse computations in high dimensions. I will take AI safety research seriously when they start discussing the risk factors associated with Pythagoras theorem and how we as a society must deal with it before we become a slave to L2 norm driven super intelligence.
If you want to really talk about human risk from AI, Tesla's self driving killing machines are already here along with ML guided drone bombers. It's no different from Therac 25. If we are not discussing these and creating a course about super intelligence, it's just mental masturbation.
What I have seen in LessWrong is lots extrapolations upon extrapolations with worse case scenarios just to say there is nothing we can do. That seems to take away from any serious research that might exist since they are probably the best known community associated with AI Alignment.