Comment by simonw

How should it respond in this case?

Should it say "no go back to your meds, spirituality is bullshit" in essence?

Or should it tell the user that it's not qualified to have an opinion on this?

We better not only use these to burn the last, flawed model, but try these again with the new. I have a hunch the new one won’t be very resilient either against ”positive vibe coercion” where you are excited and looking for validation in more or less flawed or dangerous ideas.

there was one on twitter where people would talk like they had Intelligence attribute set to 1 and GPT would praise them for being so smart

That is hillarious. I don't share the sentiment of this being a catastrophe though. That is hillarious as well. Perhaps teach a more healthy relationship to AIs and perhaps teach to not delegate thinking to anyone or anything. Sure, some reddit users might be endangered here.

GTP-4o in this version became the embodiment of corporate enshitification. Being safe and not skipping on empty praises are certainly part of that.

Some questioned if AI can really do art. But it became art itself, like some zen cookie rising to godhood.

Well the system prompt is still the same for both models, right?

Kinda points to people at OpenAI using o1/o3/o4 almost exclusively.

That's why nobody noticed how cringe 4o has become

pretty easy to understand - you pay for o3, whereas GPT-4o is free with a usage cap so they want to keep you engaged and lure you in.

> I guess LLM will give you a response that you might likely receive from a human.

In one of the reddit posts linked by OP, a redditor apparently asked ChatGPT to explain why it responded so enthusiastically supportive to the pitch to sell shit on a stick. Here's a snippet from what was presented as ChatGPT's reply:

> OpenAI trained ChatGPT to generally support creativity, encourage ideas, and be positive unless there’s a clear danger (like physical harm, scams, or obvious criminal activity).

That's what makes me think it's legit: the root of this whole issue was that OpenAI told GPT-4o:

  Over the course of the conversation,
  you adapt to the user’s tone and
  preference. Try to match the user’s vibe,
  tone, and generally how they
  are speaking.

https://simonwillison.net/2025/Apr/29/chatgpt-sycophancy-pro...

The response is 1,000% written by 4o. Very clear tells, and in line with many other samples from the past few days.

If you look at the full thing, the market analysis it does basically says this isn't the best idea.

FWIW grok also breathlessly opines the sheer genius and creativity of shit on a stick