Comment by msp26
1 year ago
I want image generators to generate what I ask them and not alter my query into something else.
It's deeply shameful that billions of dollars and the hard work of incredibly smart people is mangled for a 'feature' that most end users don't even want and can't turn off.
This is not a one off, it keeps happening with generative AI all the time. Silent prompt injections are visible for now with jailbreaks but who knows what level of stupidity goes on during training?
Look at this example from the Würstchen paper (which stable cascade is based on):
>This work uses the LAION 5-B dataset...
>As an additional precaution, we aggressively filter the dataset to 1.76% of its original size, to reduce the risk of harmful content being accidentally present (see Appendix G).
> Silent prompt injections
That’s the crux of what’s so off-putting about this whole thing. If Google or OpenAI told you your query was to be prepended with XYZ instructions, you could calibrate your expectations correctly. But they don’t want you to know they’re doing that.
Not to be overly cynical, but this seems like it's the likely outcome in the medium-term.
Billions of dollars worth of data and manhours could only be justified for something that could turn a profit, and the obvious way an advertising company like Google could make money off a prompt handler like this would be "sponsored" prompts. (i.e. if I ask for images of Ben Franklin and Coke was bidding, then here's Ben Franklin drinking a refreshing diet coke)
This sounds bit entitled. It is just service of private company.
If it's not going to give you what it's promising, which is generating images based on the prompts you provide it, it's a poor service. I think it might make more sense to try determine whether it's appropriate or not to inject ethnic or gender diversity into the prompt, rather than doing so without regard for context. I'm not categorically opposed to compensating for biases in the training data, but this was done very clumsily at best.
Yes, and I want the services I buy from private companies to do certain things.
Is it equally entitled to ask for a search engine which brings answers related to my query?