Comment by throwaway_bad | Hacker News Reader

Comment by throwaway_bad | Hacker News Reader

Metrics are really meant as a social tool for humans.

For example a commonly repeated advice is that you need a "north star" metric. For facebook this was daily active users (for an arbitrary definition of active). For machine translation research this was the BLEU score (which is also fairly arbitrary and flawed).

Humans need these simplistic metrics because we can't agree on progress otherwise.

But it's also because every human has conflicting goals that we know that the guiding metric won't fail too horribly. At all points in time, people are also evaluating progress in terms of their personal values from their point of view. Gaming of the metrics won't go unnoticed by another human. Edge cases can be patched. Conflicting goals are surfaced, decided on and resolved. The guiding metric and real mission of the organization is evolved organically. All of this is done using biological intelligence.

I don't know if it's really possible to build an AI system with the same property. The course correcting of metrics might for a long time remain in the domain of vigilant humans, as we are the only ones who know what our values are.

Metrics are really meant as a social tool for humans.

This is such an odd statement. I think the article makes a good case that in this context, metrics serve as something like the opposite of a tool - a trap, something a person or group uses that takes them something other than where they intended. Now approximation, generally, is something like a tool but it can be used well or badly.

Now, a bureaucrat or someone who want to "just get stuff done" may use a metric to get people moving without caring exactly where. But that's still with a human context, not an AI. In an AI context, a metric can run a plane into the ground or more or less anything.

IE, it's only some of the contexts (racist AI) where metrics have a social content.

The point about corner cases is reasonable but it's easy to by analogy that human correction can't keep with the problems metric yield for complex spaces; just consider how a ten dimensional cube has 2^10 literal corners and AI more or less operate in spaces with many more dimensions (of course not all "corner cases" are literal corners but there's a similar problem of intersecting "areas of concern" or whatever one would want to call them).

To clarify, I am trying to say that the only purpose of metrics is for humans in a social context. If an AI optimizing for a metric ends up with unintended consequences, that's a human problem. A human has to go in and redefine a new metric to fix it. There's no way other way around it. The human can't anthropomorphize and blame the machine. A machine can't redefine itself because it doesn't understand human values.

So it's important to understand why humans believe in metrics. Humans by default are multi-objective. We don't care about just one or even a few things, we care about a lot of things, all at once, with nobody agreeing on what they are. Identifying the "true direction" we want to go as a collective is impossible. So we pick a good enough direction so everyone can be somewhat aligned. To make it possible to communicate this vision, we compress the world down to few dimensions even though we know that it is absurd. This works fine for humans because our "distributed system" is composed of beings with enough intelligence to notice/alert/convince the rest of the system whenever we are veering off course.

Whenever we program AI with these metrics, we forget that we can't direct them we way we direct our sentient brethrens. Any simple metrics are necessarily doomed to fail because it doesn't have this self aligning property. So ultimately it is the programmer's responsibly to go and realign the system back with human values. If necessary, with some social pressure (hence this article). It doesn't seem like there will be any other way.

Your comment made me think of the Vietnam War. Many of the military leaders in the war came of age during World War II and Korea, where the most obvious metric was land: if Nazis or communists were there, then bad. If allies, then good.

But that completely broke down in Vietnam, when the placid villagers in South Vietnam by day became the ruthless Viet Cong by night. American generals had no idea how to approximate their success like they had in previous wars, so they settled on kill scores, and as long as more Vietnamese died than Americans, the DoD could go on television and claim that America was “winning”, even though the incentives for kill scores resulted in a lot of bystander villages being torched and civilians being killed, which only further put public opinion against the Americans.

I wonder if a sufficiently advanced AI system could correct its use of (or abandon) a bad metric even when it is incentivized to use it.

The problem I cited above was effectively so hard for the people at the time that it was effectively intractable. I’m not arguing that an AI would have had to solve the problem (setting aside the likelihood that even had an AI come up with a better solution other inertial forces like politics and sunk cost would have prevented the solution from being implemented), but any system that would claim to be a “strong” AI, it would at least have to be up to the task of trying.

The problem is the entire "value system" of an AI is extremely likely to be designed around metrics to begin with.

How could it discard something that it solely values?

AI, even Diamond Hard AI is independent from what it values, intelligence has no bearing on it, consider humans that are extremely intelligent but highly value brutally murdering and torturing other people. No amount of intelligence is going to change the fact they value that.

Vietnam was worse than that. There was no criteria for winning and no visible end, so the conduct of the conflict was insane.

Kill counts were an objective measurement that demonstrated that something was done, and turned into evidence of “victory“.

Intelligence, whether human or artificial, cannot fix problems that cannot be defined.

Reply on news.ycombinator.com