This is a very naive question, but considering the artifacts you have on the GAN-generated images, you are trying to generate the whole "palette image" at a certain resolution.
For which reasons did you not use a 5x1 pixel RGB image as the target, that would have been orders of magnitude faster to train?
Thanks!