But the reward (i.e., meaning) is defined by a human engineer. The point of the article is to say that we don’t have AGI yet because today’s AI needs humans to ascribe meaning to its decisions.
1. This "desire" applies only to the training algorithm. The deployed AI (running in inference) is no longer trying to minimize a loss function or maximize discounted rewards.
2. This "desire" is not a "deliberate desire". It is more like an "animal desire" such as wanting to eat because of hunger. Aristotle draws this distinction. Such a desire is automatic (rather than deliberate) and is not part of choice-making.
Many of our higher desires are set for us (taught) by other humans: our parents and society. We wouldn't want an AGI that was able to set all of its own desires, any more than we'd want that in our children. Rather dangerous indeed.