"We often cannot certify that an AI agent is optimal or ethical by looking at its source code, any more than we can certify that humans are good by scanning their brains."
What do people make of this claim? Is that a fair comparison?
What do people make of this claim? Is that a fair comparison?
Even with simpler algorithms, you might argue that in a complex environment they could have some emergent behavior that's hard to foresee based on the source code. For example, if one is not careful then even simple linear regression on some datasets can "learn" techniques like redlining -- using zip code as a proxy for race as a part of biased predictions.