Specification gaming: the flip side of AI ingenuity
deepmind.com
deepmind.com
https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3Hs...
It did this by sending its single starting unit to attack the opponent's single unit, before either could use theirs to build more.
Two identical, weak units fighting was just a coin flip.
https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden...
This is the fundamental problem. That is not the goal. As long as people believe that is the goal we will not advance.
Put another way, "the given objective" to date has always been wrong. All of the desired behavior that we are trying to get machines to emulate is a tiny subset of the learned behaviors of the capable agents (us).
The primary object we have is to survive. On a moment to moment basis our object is to fulfill a set of competing needs (hunger, tiredness, boredom etc). It is the interplay of these multiple objectives with an complex environment which necessitates the kind of intelligence that we have.
And to be absolutely clear the answer to "How do we faithfully capture the human concept of a given task in a reward function?" is "You don't" and trying to do so is harmful to the research effort.
The correct path is to evolve a reward system which functions to increase the frequencies of behaviors which meet the needs of agents in the moment and fulfill the general goal of survival. Then through a curriculum of environmental challenges* allow for those needs to only be met by accomplishing certain tasks which have value, or are of interest, to us.
*Simulator fidelity / hacking will continue to be a problem, but it is orthogonal to the problem of RL. RL finds the bugs, you fix them.
Only if you like grey goo.
Or, in a similar vein, I remember reading about how some tree drops vast numbers of acidic leaves, because its strategy for getting ahead in life is to bury competing seedlings in toxic waste.
This article has wonderful real-world examples of AIs gaming the mathematical problem specification created by the AI researcher in order to both produce a world-class score on a given dataset and be utterly unusable in the real world.
While it doesn't appear to be mentioned in the article, this is exactly why I believe that AI will not have a democratizing effect.
Either you have the mathematical and stochastic skills to describe your problem criteria as a differentiable function, or not. In the first case, you're already a good programmer, but AI might save you a bit of time over trying to find the optimal parameters manually. In the second case, you will not be able to ensure that the AI follows the implicit rules of the real world, which means that you will produce something that scores extremely well on its training data, but does not generalize to the real world.
And that's also why if someone claims a new "state of the art result on benchmark XY", you should mentally append " ... but it works nowhere else."
AI might be best put to use for finding loopholes. In formal specifications, in legal documents, in laws... you name it.
The required process change is reasonably straightforward; instead of presuming an answer given by AI is useful as a solution to the given problem, take it as an indicator of a loophole in problem specification and see if one can be closed up.
Courts are of course have a certain degree of interpretive freedom, but the process should result in a machinery that implements intent faithfully, and legislators change laws to change intent (and to patch holes, but in common law that usually happens through the courts too).
Legislation, laws, and law enforcement as technologies have always had these trade offs, and they have progressed tremendously over the centuries (or millennias).
From the Roman veto of the tribune of the plebs, to current constitutional review there are many checks and balances. But legislation as a process doesn't have inherent loopholes.
Currently for every piece of law proposed many official and civil groups warn of the potential unintended side effects of it. But then the majority usually disregards those concerns.
This. The use of "gaming" to describe the observed behaviour seems strange. To "game" suggests some malfeasance in the algorithm; a deliberate effort to subvert the intended goal, which in turn implies knowing what the "real" goal was - and somehow deciding to shortcut it.
That isn't the case. Rather, the problem lies in appropriately specifying the desired outcome, and any relevant constraints that must be obeyed. Indeed, DeepMind has separately written about the "specification problem" elsewhere [0].
That's in no way to downplay the difficulty in specification. It's one we've wrestled with in software for decades.
> Either you have the mathematical and stochastic skills to describe your problem criteria as a differentiable function, or not.
Indeed. That statement could equally have been written ~30 years ago when Z, VDM and the likes were active areas of research. More recently, TLA+ has gotten some traction, but it's still very much in the margins.
Perhaps ML will spur the next wave of specification formalisms. But it will face the same challenge as its pre-decessors: how to make it accessible to more than a very (very) small set of sufficiently capable few.
[0] https://medium.com/@deepmindsafetyresearch/building-safe-art...
We could be just around the corner, but the fact that people are taking for granted that there's no intelligence in AI isn't promising.
If you respond defensively about how difficult it is to give people what they want, well, that's the whole point.
For example, people were willing to declare optical flow "solved" already 5 years ago. Yet, autonomous drones still crash into wires, glass, trees, snow, water, etc., because their optical flow algorithms cannot handle these aspects of reality (which were not present in the AI training data).
https://en.wikipedia.org/wiki/Mechanism_design
The objective is designed as part of the game, but it's very difficult to get the incentives right.
It is a pain point in every system that tries to solve a problem through generative meta-heuristic search. Translating intention into a set of explicit constraints that separate desired outcomes from undesired ones that satisfy the letter of the constraints but not the intent gets exponentially harder as the dimensionality of the solution space grows.
What's the loss function for this? How do you cross-validate this loss? The only generally applicable methods I can think of (essentially some Bayesian-type updating, which then uses Monte-Carlo-type methods for evaluating this loss and "uncertainty") is just so much more expensive than anything we do at the moment that, at the current computing level, we wouldn't even be able to do it in most cases.
The problem is, an AI model will tell you an "uncertainty" but these "uncertainties" will often not reflect the true underlying probability that it is wrong. This is, of course, all a bit silly since it's hard knowing what you don't know, and a proper update to this number is also not well-defined unless you already have a bunch of samples about potential future outcomes (via, like I mentioned before, a Monte-Carlo type approach. This would multiply the cost of something as simple as an evaluation by many times.)
The measure of uncertainty could be as simple as a multi armed bandit where the reward is % of accurate guesses following that path.
The pseudo adversarial aspect of this being that both models will learn to agree on how choices translate to gameplay changes. The bot will not learn to beat the game necessarily, but a Style GANN kind of approach would probably be just fine to saying, "Please get to a state that looks like this" with a few manual samples. Perhaps all of this to say, it's better to learn how to play the game, and specify that, in this particular run, you'd like the bot to try and reach the game end credits.
[1] Model-Based Active Exploration, Shyam et al. 2019, https://arxiv.org/abs/1810.12162
In the extreme case, specification gaming is the exact opposite: an AI will do anything in its power to satisfy the reward function without having to do any work, including tricking its creator into making the reward function easier
Human students and workers need a fair amount of "work specifications" (years of teaching, decades of workplace laws and safety measure, etc) to actually complete the requested work that allows them to have revenue/salary. I don't see the specification game as an AI specific issue.
Who knew?