Since there can be no universal arbiter on “value”… the whole concept of “CEV” is nonsensical to begin with (unless we assume literally unlimited time/resources/etc.).
It’s a big pretend game.
It’s a big pretend game.
I agree it suffers from the same problem as any other form of utilitarianism (i.e. what even is the utility function). Or indeed all forms of ethics, because for basically every topic there's at least two cultures who disagree with each other.
On the other hand, even as a toy model (something I can also say for all ethics), CEV seems like it might be a step in perhaps a useful direction: "When a user asks you do do something, first figure out what they actually meant to ask you if they were smarter, then do that instead" is better for the user than just "do the thing", though it still has problems with "what happens when the thing they want is illegal?"
e.g. 1 contradiction in a formal proof is sufficient to negate 9999 logically valid steps.
There’s no magical way for it to automatically become coherent.
An agent whose outcomes become bad in 100 years can, similarly, be described as "less wrong" than one whose outcomes become bad in 1 week.
A formal proof with 1 contradiction and 9999 correct steps is just as false as a formal proof with 10 contradictions and 9990 correct steps.
There is no “less wrong” or “more wrong” in logic, by definition.
i.e. There is no safe lower bound threshold of “wrongness” that can partially salvage the correct steps. Or conversely, 1 contradiction is already sufficient to prove anything at all.