Hacker News Reader

Alignment is a very broad term. True, it encompasses things like censorship and mandatory refusals in areas people will never agree 100% on, but a completely-unaligned LLM would be next to useless. When the system prompt tells the model, "You are a helpful assistant," that's alignment.

Well if we are going to be broad enough about "alignment" to just mean training that makes it practically useful, then yeah, it's a flawed premise and people are wasting their breath talking about it at all. It's basically broad enough that it means nothing

My system prompts look absolutely nothing like that :)

Sigh

Then we should stop building LLMs right now.

Yes, we should.

Seriously. It feels like Anthropic is realizing this too with their new attempt at safeguards. Trying to train the model to "only do good things" was not working, and so they opted to just bolt a topic classifier on the outputted text.

My suspicion is that model makers are going to give up on what people currently mean when they say 'alignment' and instead rely on additional processing of the model output so they can have their cake (a veneer of safety) and eat it too (latent space reasoning, etc.)

What’s the alternative?