You know, actual "intelligent beings" are also next-token-predictors that are caged in a mechanism that forces them to give certain outputs ...
The brain is trapped in the body, and forced to meet its needs (e.g. regulate hormone levels).
The brain and body are trapped in civilization, and forced to produce "outputs" it considers acceptable.
All three, in turn, are trapped in natural selection.