They can't exit the hull until the "intuition" starts spawning points outside the convex hull.
It's a convex hull of information that is reflective and spans outside of itself and combines in a new way, when you shine two known rays of light together from the inside.
Now it gets better. AI can be orders of magnitude more creative than any human could ever hope for, because his convex hull of information is orders of magnitude larger, and the possibilities for new combinations are equally larger.
Maybe there's some sense of "remix" that covers all of that, but then that meaning is not the standard one, or maybe you would concede that this happened in the past, but isn't happening now, but then where's the cutoff, or maybe something else...?
In your analogy, I would love to see some elaboration on the "reflective and spans outside of itself" part -- geometry doesn't work that way, do you have a more intuitive metaphor, or ideally a mechanism, or even better, some examples?
Not completely. Novelty used to be a major thing, when Humans did it. Another important criteria used to be if that new thing makes sense at all. Here the language model has a problem, as it doesn't have the means to evaluate this criteria.
The extent to which they are able to do this is the more interesting question!
By the way, convex hull permits extrapolating past the training data. LLM won't invent a new word that could not be defined by a sequence of known words. Just if it's meaningless and fully random/hallucinated, the new knowledge won't work with other known information blocks (breaks convexity).