Nature itself plans trajectories to minimize action, the integral of a Lagrangian: it's literally just a number, but it decides the motion of molecules and galaxies. What's wrong with a moral system based around minimizing a pain integral, subject to constraints? Still just a number. But one that can easily take an important position in our decision-making, and find a deep-seated place in our minds as well.
Of course, we probably ended up with morals in the first place as a distillation of game-theoretic heuristics, which helped us trust and cooperate with each other, which would have been an evolutionary benefit. But as with anything else, we're accountable to the drives that evolution left us with, not evolution's reasons for putting them there. The morals are in our minds, not in nature, but either way it's numbers all the way down.