Read up about the "alignment problem". It is one that ai researchers cannot Crack and its literally what you wrote. The idea that we could never in any way define goals align with us.
I suspect we'll hear of Dr. Susan Calvin more and more in the future.