>>But if they do make a point it's that we don't need to (and often can't) understand what will result from systems that have a built in feedback mechanism.
That may be true, but my point was more to the effect that there are things we fail to understand about human nature that drives decision making, actions, and goal setting within our own society, let alone other cultures. Consider how factors such as your own personal energy levels affect your decision making throughout the day (decision fatigue)...how are we going to model something like that with AI? Or consider that many different cultures around the world have very different societal goals than Americans do pursuing personal freedom and material well-being. How do we consider those different abstract goals, that most citizens don't even realize or understand why they pursue, because it's just how they have been raised. How do environmental factors drive decision making in different climates? Those who live in a very cold climate have to take different precautions and deal with different risks than those in warm climates do. What about locally available resources? Many business (and design) decisions are made based as much upon supply chain availability and risk management as they are customer demand. How does AI consider these factors when AI doesn't even know these things exist? These are the things I was talking about that we still have difficulty understanding about each other, let alone being able to program a robot to imitate.