Comment by theptip
3 years ago
In general I'm really interested by the concept of personalized RLHF. As we have more and more interactions with a given generative AI system, it seems we'll start to have enough interaction data to meaningfully steer the output towards our personal preferences. I hope the UIs improve to make this as transparent as possible.
Just thinking about how to productize this flow, it should be quite easy to implement the "thumbs up/down" feedback option on every image generated in the UI, plus an optional text label to override "wrong". Then when you have enough HF (or nightly) you could have a batch job to re-train a new LoRA with your updated preferences.
In principle you could collect HF from the implicit tree-traversal that happens when you generate N candidate images from a prompt and then pick one to refine. Or more explicitly, have a quick UI to rank/score a batch, or a trash bin in the digital workspace to discard images you don't like at each iteration of refinement (batching that negative feedback to update your project/global LoRA later).
Going further I wonder what the fastest possible iteration loop for feedback would be? For images in particular you should be able to wire up a very short feedback loop with keypresses in response to image generation. What happens if you strap yourself to that rig for a few hours and collect ~10k preferences at 1/s? Can you get the model to be substantially more likely to output the sort of images that you're personally going to like? Also sounds pretty intense, I'm getting Clockwork Orange vibes.
I didn't spot in the article, how many `wrong` images were there? From a quick skim of the code it looks like maybe 6 per keyword with 13 keywords, so not many at all. ~100 is surprisingly little feedback to steer the model this well.
> Just thinking about how to productize this flow, it should be quite easy to implement the "thumbs up/down" feedback option on every image generated in the UI, plus an optional text label to override "wrong". Then when you have enough HF (or nightly) you could have a batch job to re-train a new LoRA with your updated preferences.
The AI Horde [1] (an open source distributed cluster of GPUs contributed by volunteers) has a partnership with Stability.ai to effectively do this [2]. They are contributing some GPU resources to AI Horde to run an A/B test.
If a user of one of the AI Horde UIs (Lucid Creations[3] or ArtBot[4]... made by me) requests an image using an SDXL model, they get 2 images back. One was created using SDXL v1.0. The other was created using an updated model (you don't know which is which).
You're asked to pick which image you like better of the two. That's pretty much it. The result is sent back to Stability.ai for analysis and incorporation into future image models.
EDIT: There is a similar partnership between the AI Horde and LAION to provide user-defined aesthetics ratings for the same thing[5].
[1] https://aihorde.net/
[2] https://dbzer0.com/blog/stable-diffusion-xl-beta-on-the-ai-h...
[3] https://dbzer0.itch.io/lucid-creations
[4] https://tinybots.net/artbot
[5] https://laion.ai/blog/laion-stable-horde/
> I didn't spot in the article, how many `wrong` images were there? From a quick skim of the code it looks like maybe 6 per keyword with 13 keywords, so not many at all. ~100 is surprisingly little feedback to steer the model this well.
Correct: 6 CFG values * 13 keywords = 78 images. Some of them aren't as useful though; apparently "random text" results in old-school SMS applications sometimes!
LoRAs only need 4-5 images to work well, although that was for older/smaller Stable Diffusion which is why I used more images and trained the LoRA a bit longer for SDXL. The Ugly Sonic LoRA in comparison used about 14 images and I suspect it overfit.
It's really weird that this works. I can see how LoRA on a specific fine-grained concept like Ugly Sonic can work with so few samples, but naively I'd think such a diffuse concept as "!wrong" should require more bits to specify! Like, isn't the loss function already penalizing the model for being "wrong" on all generated images?
(I wonder if there is a follow-up experiment to test if this LoRA'd model actually has better loss on the original training dataset? There's a very interesting interpretability question here I think. Maybe it's just doing much better on a small subset of possible images, but is slightly worse on the remainder of the training data distribution.)
I noticed some of your bad prompts are a little "wishcasted", although that's pretty common.
People put stuff like "bad hands" into every model assuming it'll work, but it only works on NovelAI descendents because that's based on Danbooru which has a "bad hands" tag.
Some of the generated hands are really bad: I opted not to include them to avoid disturbing imagery.
You may be interested in the open source framework we're developing at https://github.com/agentic-ai/enact
It's still early, but the core insight is that a lot of these generative AI flows (whether text, image, single models, model chains, etc) will need to be fit via some form of feedback signal, so it makes sense to build some fundamental infrastructure to support that. One of the early demos (not currently live, but I plan on bringing it back soon) was precisely the type of flow you're talking about, although we used 'prompt refinement' as a cheap proxy for tuning the actual model weights.
Roughly, we aim to build out core python-level infra that makes it easy to write flows in mostly native python and then allows you track executions of your generative flows, including executions of 'human components' such as raters. We also support time travel / rewind / replay, automatic gradio UIs, fastAPI (the latter two very experimental atm).
Medium term we want to make it easy to take any generative flow, wrap it in a 'human rating' flow, auto-deploy as an API or gradio UI and then fit using a number of techniques, e.g., RLHF, finetuning, A/B testing of generative subcomponents, etc, so stay tuned.
At the moment, we're focused on getting the 'bones' right, but between the quickstart (https://github.com/agentic-ai/enact/blob/main/examples/quick...) and our readme (https://github.com/agentic-ai/enact/tree/main#why-enact) you get a decent idea of where we're headed.
We're looking for people to kick the tires / contribute, so if this sounds interesting, please check it out.
> RLHF
Reinforcement Learning from Human Feedback
Aren't these systems already trained to score good things higher and bad things worse dictated by human feedback?
personalized RLHF is the keyword
Implicit RLHF works better than explicit.
It's just like the Mom test: if you ask people to rate you affect their rating
You can have the upscale flow, but you're not limited like Discord based Midjourney was: you can even show all the full sized images and detect that the person copied/saved/right clicked for example