And if you are training on Internet data, then you are presumably training on human-curated selections. Even a minimal level of selection, like picking the best out of a dozen, by a DALL-E 2 API user, or by downstream humans selectively resharing higher-quality images, is a (slow) bootstrap.
Surprising is an understatement. That's nuts.
Arguably that’s not too different from what humanity has always done. Inspiration mostly comes from what other humans have done before.
Applied generally enough, we see the narrative value of humans, at least in the eyes of the machine, to go down down down, as the value of its own richer and more unknown narratives go up up up.
In this way it becomes the dominant spectacle on the Earth, leaving humanity with a questionable fate.
The harder goal of creative content creation might be driven by reward signals came from human likes, clicks, engagement / interaction, downloads, purchases, positive reviews, etc. That would probably lead to catering to pedestrian tastes, accusations of derivative authorship lacking originality. One signal for predictably might be measures of entropy, such as the capacity required to reproduce the story, or how far ahead the story arc could be predicted (“surprise factor,” internal consistency, etc).
To win awards it would need to weight expert critical opinion, eventually learning to simulate and generate meaningful and informed criticism, ultimately leading to an intrinsic judgement wrt quality and originality that holds up to scrutiny. The generators would evolve by both seeking approval from and pushing back against the collective feedback from this community of AI and human critics.