Comment by minimaxir
3 years ago
> I didn't spot in the article, how many `wrong` images were there? From a quick skim of the code it looks like maybe 6 per keyword with 13 keywords, so not many at all. ~100 is surprisingly little feedback to steer the model this well.
Correct: 6 CFG values * 13 keywords = 78 images. Some of them aren't as useful though; apparently "random text" results in old-school SMS applications sometimes!
LoRAs only need 4-5 images to work well, although that was for older/smaller Stable Diffusion which is why I used more images and trained the LoRA a bit longer for SDXL. The Ugly Sonic LoRA in comparison used about 14 images and I suspect it overfit.
It's really weird that this works. I can see how LoRA on a specific fine-grained concept like Ugly Sonic can work with so few samples, but naively I'd think such a diffuse concept as "!wrong" should require more bits to specify! Like, isn't the loss function already penalizing the model for being "wrong" on all generated images?
(I wonder if there is a follow-up experiment to test if this LoRA'd model actually has better loss on the original training dataset? There's a very interesting interpretability question here I think. Maybe it's just doing much better on a small subset of possible images, but is slightly worse on the remainder of the training data distribution.)
I noticed some of your bad prompts are a little "wishcasted", although that's pretty common.
People put stuff like "bad hands" into every model assuming it'll work, but it only works on NovelAI descendents because that's based on Danbooru which has a "bad hands" tag.
Some of the generated hands are really bad: I opted not to include them to avoid disturbing imagery.