>My original statement should have been: "We can't model Humanity's Collective Objective Function" - which is what would be behind what we are interested in: Stable functioning muti-agent systems. I think EY took a crack at this a long time ago and rightly abandoned the concept (see: CEV).
If no such thing exists, then it was the wrong thing to investigate, so stop being interested in it.
>Even with that clarification I disagree with the premise that we can model an "objective function" for an individual strictly in-vivo. Modelling an individual agent's reasoning/function system doesn't account for the environmental context it exists inside of, gives input into and responds to. So even if it was possible to understand the mechanism for intra-personal decision criteria, and I don't think it probably is, I don't think it's generalizable without having the context of inputs.
That's just an inverse reasoning/theory-of-mind problem, one that normal theory-of-mind models and actual human brains solve every day.
>Assuming that we could do this, I don't think you can extrapolate intentionality directly from individual to collective groups - which for an AGI is what is existentially important as it needs to be collectively general to solve the existential problem.
What's this about "collectively general" and "the existential problem"? You seem to have gone off the deep end into philosophy salad.
>I also don't think this is desirable as a framework for AGI - as humans, despite our intelligent status, are quite unstable and sub-optimal in groups.
Considering you don't seem to know much about how humans work and what causes us to work well or badly in various situations, this statement comes off as almost racist.