Hacker News Reader

Comment by rabuse | Hacker News Reader

Creating art with stable diffusion has become such a fun hobby of mine. The difference between SD 1.5/2.0 and SDXL is massive, and it's impressive how quickly the quality is improving with this stuff.

View on HN

>The difference between SD 1.5/2.0 and SDXL is massive,

Can you explain?

I havent used SDXL yet, but I spent a ton of time in 1.5.

So far I gathered:

>Higher res

>higher 'quality'

But given I was using realistic vision 3 for so long, I never had a quality issue. With upscaling, I never needed higher res.

Yes, currently SDXL doesn't really beat the best SD1.5 checkpoints quality-wise. But it (and the currently available checkpoints) shows awesome promise, so give it a six months or so.

The best 1.5 checkpoints are constrained in their output flexibility to achieve the quality they get though, and they don't follow prompts nearly as well as SDXL, so if the model doesn't naturally gravitate towards doing what you want it's very hard to steer it anywhere. SDXL also does a better job with full anatomy, which is the reason shared 1.5 generations tend to be torso up or portrait shots.

Currently SDXL is better than SD1.5 checkpoints at pretty much everything other than portraits (or anime drawings) of pretty women.

Unfortunately it seems that's all people want to generate, as is evident when you search for SD on Twitter.

Stable diffusion doesn’t grant the user a good imagination or taste unfortunately

Yes, point conceded, I should've said something about the flexibility and capability of SDXL rather than just image quality in a narrow sense.

From my experiments it seems that SD XL understands prompts much better. While SD 1.5 is great at generating your typical "anime girl with big boobs" stuff - if you try to generate something a little bit more unusual - it usually doesn't generate exactly what you want and seems to straight up ignore large parts of the prompt.

SD XL seems to understand weird and unusual prompts a lot better.

SD XL is capable of generating 1024x1024 images without hacks like "hires fix". That's a very good thing, because hires fix sometimes introduces additional glitches while upscaling. Especially at higher denoising strength. Hires fix fixed the broken face - yay, but the subject now has 3 legs instead of two. Things like that happen far less often with SD XL.

> While SD 1.5 is great at generating your typical "anime girl with big boobs" stuff - if you try to generate something a little bit more unusual - it usually doesn't generate exactly what you want and seems to straight up ignore large parts of the prompt.

Pretty much experience with SD 1.5, but I'll give XL a try.

I hope you'll forgive me for a bit of a self promotion here, but I think I have an interesting example of SD 1.5 (what most people are familiar with and what most models are based off of) vs SDXL.

Before Phony Stark shut down the Twitter API, I was running a bot that created landscape images with Stable Diffusion v1.5. Its name is Mr. RossBot [1]. Check out the Twitter page for some examples of the quality.

This weekend, I finally updated the code to get it running on Mastodon. In the process, I updated the model to use SDXL [2]. It's running the exact same code otherwise to randomly generate prompts.

The image caption is a simplified version of the prompt. e.g., "Snowcapped mountain peaks with an oxbow lake at golden hour."

Behind the scenes, a whole bunch of extra descriptive stuff is added, so the prompt that SD v1.5 / SDXL get is: "beautiful painting of snowcapped mountain peaks with an oxbow lake at golden hour, concept art, trending on artstation, 8k, very sharp, extremely detailed, volumetric, beautiful lighting, serene, oil painting, wet-on-wet brush strokes, bob ross style"

Anyway, I feel like the quality of SDXL is sharper and it just nails subjects a lot better. It also tries to add reflections and shadows (not always correctly), whereas that didn't happen as much with SD v1.5.

I'm pretty impressed! Especially because Stability.ai had released an update model of Stable Diffusion before SDXL: SD v2.0 and SD v2.1. The results (IMHO) were absolute garbage using the same prompts.

[1] https://twitter.com/mrrossbot

[2] https://botsin.space/@MrRossBot

Here's an example using my dog - a trained checkpoint on one of the nicer SD 1.5 models and a LoRA for the SDXL ones: https://imgur.com/a/PklEKwC

The first 3 images are some of my attempts at making her into a Pokemon. Some turned out pretty good (after generating 50+ per type), but I struggled with water in particular. It was hard to get her to have a fin, especially with no additional tail.

I haven't done many in SDXL, but that's the point. I've probably generated..10 images of her as a Pokemon, just when I first trying out the LoRA. The next 2 images are from that, and that was before I had a good ComfyUI workflow to boot.

The rest are various sample images from SDXL showing how versatile it is. In most of those, I only had to generate a few images per prompt to get something pretty darn great. In the Halo 2 one the prompt was literally "an xbox 360 screenshot of cinderdog in Halo 2, multiplayer."

And it made her into a freaking Elite, and it worked wonderfully. I previously tried to generate ones like those candyland images in 1.5 models and the foreground and background just didn't look good. In SDXL it just works.

Very cool! How many images did you use to create the LoRa of your dog? Do you have any guide to recommend?

It was about 30 images, though I'm planning on adding more and training again sometime. Either that or splitting it up between when her hair is short and when it's long, as it really changes how she looks.

This isn't what I used for my dog's LoRa but I used it for my wife and it worked better than what I was doing before (Adafactor): https://civitai.notion.site/SDXL-1-0-Training-Overview-4fb03...

I'd recommend increasing the network dimension to at least 64, if your VRAM can take it. I can do 64 with my 12GB card. At least for people, I've had better luck using a token that's a celebrity. I'm not sure how to try that with my dog - perhaps just "terrier dog" or something.

Thanks! Looks like I'll need to rent a GPU to use SDXL fine tuning. Poor old RTX2060 not gonna cut it.

That's a very low learning rate -- between 2-3 orders of magnitude lower than what I've seen for that number of steps. I'll have to give it a try.

I should have been clear - I'm using the Prodigy settings on that page, not the Adafactor one. You set the learning rate to 1 and the scheduler to cosine, but the real learning rate is figured out by the optimizer.

For simplicity, it feels like SDXL has better "defaults". You don't have to include a bunch boilerplate keywords to wrangle it into generating good images.

The flip side is I've found it a bit harder to tweak prompts

I've found it very hard to create different styles with SDXL. If you want photorealism, anime, sci-fi, or somewhere in between, it's amazing.

But I've been trying to get it to generate equivalent quality in other styles, e.g. watercolor, abstract painting etc. It doesn't seem to be easy - the quality drops a lot and it's harder to avoid weird results like people wearing enormous hats or distorted perspective.

Admittedly I haven't spent a huge amount of time on this because generation is just a bit too slow to be enjoyable on my machine. Has anyone else had success here?

Reply on news.ycombinator.com