Comment by mrandish
2 days ago
Over hours of experimentation with various LLMs, I've found virtually any system prompt can cause unintended skewing of the model's output. Even just 5 to 8 short, direct words about length, tone or formatting can cause subtle yet significant changes in model output.
Longer, more detailed or conditional prompts always introduce an additional cognitive load as it checks every token it generates against the conditions. Making instructions more absolute (like: "Never do...") can increase the duration of compliance but at the cost of creating a significant center of attentional gravity. This can cause far more output distortion as the model devotes increasing portions of its attention budget to ensure compliance with a heavyweight requirement or prohibition. Every word in a global prompt is a trade-off between attention, compliance, drift, etc.
As someone used to thinking of computers as natural deterministic rule-followers, it's weird having to carefully wordsmith and A/B test even the simplest global prompts. It feels like coaxing a hyper-literal, emotionally sensitive, spectrum-ish toddler to comply but without being so strict it gets 'upset' or spirals into hyper-focusing.
True. The real trick, if you have a client-side agent framework to hand, is to prompt it once as "gently" as possible to "just solve the problem"; and then, after its response to that, automatically prompt it again, with a separate prompt, to summarize that response a certain way. That way, the second prompt isn't "in mind" during generation of the first prompt. (And ideally, you don't even present the intermediate result to the user.)
Sadly, you can't do things like this directly using ChatGPT's own "GPTs" abstraction. (For that feature to be useful, they really need some concept of server-side agents as stateful resident IO-stream-reducer actors.)
Models can be so sensitive that even prompting "Number section headings" would cause it to stop using its normal bullet point formatting anywhere. But then adding some variant of "...but don't stop using bullets as you normally do when they are needed" would make it start using bullets all the time.
Trying to craft a workable prompt got so frustrating I eventually just tried a prompt of "Don't change anything about your normal text formatting, it's perfect as is" and even that skewed the output vs no prompt. For browser chat I finally just wrote a client-side CSS UserStyle that does the formatting. Now I even have sequentially numbered sections with indented alphabetic bullets! Zero cognitive load or attentional skew and it never drifts off the formatting in long sessions.
[flagged]