Generative models like GANs are fascinating, but very temperamental to train. Some of the findings in this paper mirror my own observations - increasing the complexity of the GAN adds a lot of instability. My solution was to keep things as simple as possible. I spent a lot of effort trying to increase the size of the network to get better results, but in the end my smallest implementation worked the best.
This bit is interesting: "Without progressive growing, all layers of the generator and discriminator are tasked with simultaneously finding succinct intermediate representations for both the large-scale variation and the small-scale detail. With progressive growing, however, the existing low-resolution layers are likely to have already converged early on, so the networks are only tasked with refining the representations by increasingly smaller-scale effects as new layers are introduce"