Also, just fyi, Google's alternative to Stable Diffusion/DALLE-2, Imagen is not CLIP guided. Thus being CLIP guided is not necessary for the problem.
Reply on news.ycombinator.com