Comment by carbocation

Tangentially related: for reasons I don't yet really understand, the LORAs that I build for Stable Diffusion XL only work well if I give a pretty generic negative prompt.

These are fine-tuned on 6 photos of my face, and if I use them with positive prompts, the generated characters don't look much like me. But if I add generic negative terms like "low quality", suddenly the depiction of my face is almost exactly right.

I've trained several models and this has been true across a range of learning rates and number of training epochs.

To me, this feels like it will somehow ultimately be connected to whatever is driving minimaxir's observations in this post.