Comment by zahlman
5 days ago
Why wouldn't the system prompt be controlled on the server side of the API? I agree with https://news.ycombinator.com/item?id=47010577 ; I think results like this more likely come from "roleplaying" (lightweight jailbreaking).
The websites and apps probably have a system prompt that tells them to be more cautious with stuff like this, so that AIs look more credible to the general public. APIs might not.