Comment by rabuse

I hope you'll forgive me for a bit of a self promotion here, but I think I have an interesting example of SD 1.5 (what most people are familiar with and what most models are based off of) vs SDXL.

Before Phony Stark shut down the Twitter API, I was running a bot that created landscape images with Stable Diffusion v1.5. Its name is Mr. RossBot [1]. Check out the Twitter page for some examples of the quality.

This weekend, I finally updated the code to get it running on Mastodon. In the process, I updated the model to use SDXL [2]. It's running the exact same code otherwise to randomly generate prompts.

The image caption is a simplified version of the prompt. e.g., "Snowcapped mountain peaks with an oxbow lake at golden hour."

Behind the scenes, a whole bunch of extra descriptive stuff is added, so the prompt that SD v1.5 / SDXL get is: "beautiful painting of snowcapped mountain peaks with an oxbow lake at golden hour, concept art, trending on artstation, 8k, very sharp, extremely detailed, volumetric, beautiful lighting, serene, oil painting, wet-on-wet brush strokes, bob ross style"

Anyway, I feel like the quality of SDXL is sharper and it just nails subjects a lot better. It also tries to add reflections and shadows (not always correctly), whereas that didn't happen as much with SD v1.5.

I'm pretty impressed! Especially because Stability.ai had released an update model of Stable Diffusion before SDXL: SD v2.0 and SD v2.1. The results (IMHO) were absolute garbage using the same prompts.

[1] https://twitter.com/mrrossbot

[2] https://botsin.space/@MrRossBot

Here's an example using my dog - a trained checkpoint on one of the nicer SD 1.5 models and a LoRA for the SDXL ones: https://imgur.com/a/PklEKwC

The first 3 images are some of my attempts at making her into a Pokemon. Some turned out pretty good (after generating 50+ per type), but I struggled with water in particular. It was hard to get her to have a fin, especially with no additional tail.

I haven't done many in SDXL, but that's the point. I've probably generated..10 images of her as a Pokemon, just when I first trying out the LoRA. The next 2 images are from that, and that was before I had a good ComfyUI workflow to boot.

The rest are various sample images from SDXL showing how versatile it is. In most of those, I only had to generate a few images per prompt to get something pretty darn great. In the Halo 2 one the prompt was literally "an xbox 360 screenshot of cinderdog in Halo 2, multiplayer."

And it made her into a freaking Elite, and it worked wonderfully. I previously tried to generate ones like those candyland images in 1.5 models and the foreground and background just didn't look good. In SDXL it just works.

From my experiments it seems that SD XL understands prompts much better. While SD 1.5 is great at generating your typical "anime girl with big boobs" stuff - if you try to generate something a little bit more unusual - it usually doesn't generate exactly what you want and seems to straight up ignore large parts of the prompt.

SD XL seems to understand weird and unusual prompts a lot better.

SD XL is capable of generating 1024x1024 images without hacks like "hires fix". That's a very good thing, because hires fix sometimes introduces additional glitches while upscaling. Especially at higher denoising strength. Hires fix fixed the broken face - yay, but the subject now has 3 legs instead of two. Things like that happen far less often with SD XL.

For simplicity, it feels like SDXL has better "defaults". You don't have to include a bunch boilerplate keywords to wrangle it into generating good images.

The flip side is I've found it a bit harder to tweak prompts

Yes, currently SDXL doesn't really beat the best SD1.5 checkpoints quality-wise. But it (and the currently available checkpoints) shows awesome promise, so give it a six months or so.