I believe that what we can reasonably expect in the future is machines that act as if they experience consciousness. But the fact that this is true consciousness is highly debatable, as the thought experiment of the Chinese room [1] explains.
The Chinese room thought experiment is obviously flawed to me. It's not the "computer" that is conscious, it's the running software.
An AI cannot be removed from its box, because it doesn't have one. It really does have enough information inside of its essence to reply. In fact, that information makes up its essence.
I agree that in some sense their knowledge is distinct and of a different character to human knowledge. But what that means conceptually or morally exactly is very complicated, and cannot be dismissed easily