I think this is an insightful question and potentially a good idea. It's common practice to generate with GAN:s in several stages, elaborating each time on an initiation which is often cruder or vaguer.

I see no reason why principles like this https://dl.acm.org/doi/10.1145/3306305.3332370

or this https://github.com/EvgenyKashin/stylegan2-distillation

couldn't be used to create much richer environments, at least in terms of media.