That is precisely the premise of the novel "Blindsight" by Peter Watts. ChatGPT and its ilk feel to me like the aliens in the novel. Extremely intelligent, but not at all conscious / sentient.