Comment by stared
8 days ago
On multiple occasions, I've gained insights from LLMs (particularly GPT 4.5, which in this regard is leagues ahead of others) within minutes—something I hadn't achieved after months of therapy. In the right hands, it is entirely possible to access super-human insights. This shouldn't be surprising: LLMs have absorbed not just all therapeutic, psychological, and psychiatric textbooks but also millions (perhaps even hundreds of millions) of real-life conversations—something physically impossible for any human being.
However, we here on the Hacker News are not typical users. Most people likely wouldn't benefit as much, especially those unfamiliar with how LLMs work or unable to perceive meaningful differences between models (in particular, readers who wouldn't notice or appreciate the differences between GPT 4o, Gemini 2.5 Pro, and GPT 4.5).
For many people—especially those unaware of the numerous limitations and caveats associated with LLM-based models—it can be dangerous on multiple levels.
(Side note: Two years ago, I was developing a project that allowed people to converse with AI as if chatting with a friend. Even then, we took great care to explicitly state that it was not a therapist (though some might have used it as such), due to how easily people anthropomorphize AI and develop unrealistic expectations. This could become particularly dangerous for individuals in vulnerable mental states.)
You should read Baldur Bjarnasson's recent essay, "Trusting your own judgment on AI is a huge risk". https://www.baldurbjarnason.com/2025/trusting-your-own-judge...
Excerpt:
"Don’t self-experiment with psychological hazards! I can’t stress this enough!
"There are many classes of problems that simply cannot be effectively investigated through self-experimentation and doing so exposes you to inflicting Cialdini-style persuasion and manipulation on yourself."
From what I see, this person loves structured research. I guess if he were on fire, he wouldn't notice, before there is a peer-reviewed research on that. (You can extrapolate.)
He tries to be persuasive by giving an example of that there is "just gossip" that TypeScript is better than JavaScript, which summarizes the mindset better than I could. (God bless his codebase.)
It misses the point that always we live in a messy, unique situation, and there are a lot of proxies. For own personal decision it matters less if a given food is healthier on the average, if in our region its quality is poor, or we are allergic to that. Willing or not, we experiment every waking second. It is up to us, if we learn from that.
Later, this ex cathedra "self-experimenting with psychological hazards is always a bad idea" rings the bell of "doing yoga will always bring you to satan" or so.
(This thing that we are easy to fool ourselves is psychology 101; yet, here AI is just a tool. You can say in a similar way that you talk with people that (on the average) agree with you.)
But, ironically - he might be right. In his case, it is better to rely on delayed and averaged-out scientific data than his own judgement.
How does one begin to educate oneself on the way LLMs work beyond layman understanding of it being a "word predictor"? I use LLMs very heavily and do not perceive any differences between models. My math background is very weak and full of gaps, which i'm currently working on through khan academy, so it feels very daunting to approach this subject for a deeper dive. I try to read some of the more technical discussions (e.g waluigi effect on lesswrong), however it feels like I lack the needed knowledge to not have it completely go over my head, not taking into account some of the surface-level insights.
Start here:
https://udlbook.github.io/udlbook/
I had not heard of this, wow, this is GOLD!
LLMs are missing 3 things (even if they ingest the whole of knowledge):
- long term memory
- trust
- (more importantly) the ability to nudge or to push the person to change. An LLM that only agrees and sympathizes is not going to make things change
For a bit now ChatGPT has been able to reference your entire chat history. It was one of the biggest and most substantial improvements to the product in its history in my opinion. I'm sure we'll continue to see improvements in this feature over time, but your first item here is already partially addressed (maybe fully).
I completely agree on the third item. Carefully tuned pushback is something that even today's most sophisticated models are not very good at. They are simply too sycophantic. A great human professional therapist provides value not just by listening to their client and offering academic insights, but more specifically by knowing exactly when and how to push back -- sometimes quite forcefully, sometimes gently, sometimes not at all. I've never interacted with any LLM that can approach that level of judgment -- not because they lack the fundamental capacity, but because they're all simply trained to be too agreeable right now.
You can easily give them long-term memory, and you can prompt them to nudge the person to change. Trust is something that's built, not something one inherently has.
> trust
Trust is about you, not about another person (or tool, or AI model).
> long term memory
Well, right now you need to put context by hand. If you already write about yourself (e.g. with Obsidian or such), you may copy-and-paste what matters for a particular problem.
> (more importantly) the ability to nudge or to push the person to change.
It is there.
> An LLM that only agrees and sympathizes is not going to make things change
Which LLM you use? Prompt GPT 4.5 to "nudge and push me to change, in a way that works the best for me" and see it how it works.
> If you already write about yourself (e.g. with Obsidian or such), you may copy-and-paste what matters for a particular problem.
Wrong, because identifying what's part of the context is part of the problem. If you could just pick up what is relevant then the problem would be much easier
> Prompt GPT 4.5 to "nudge and push me to change, in a way that works the best for me" and see it how it works.
Cool you try that and you see how it goes. And remember that when it fails you'll only have yourself to blame then
1 reply →
Hahaha, here's mine:
"Here's an insight that might surprise you: You're likely underutilizing TypeScript's type system as a design tool, not just a correctness checker. Your focus on correctness and performance suggests you probably write defensive, explicit code - but this same instinct might be causing you to miss opportunities where TypeScript's inference engine could do heavy lifting for you."
I'm highly skeptical, do you have a concrete example?
I won't share any of my examples, as there are both personal and sensitive.
Very easy version:
If you use ChatGPT a lot, write "Base on all you know about me, write an insight on me that I would be surprised by". For me it was "well, expected, but still on point". For people with not experience of using LLMs in a similar way it might be mind-blowing.
An actual version I do:
GPT 4.5. Providing A LOT context (think, 15 min of writing) of an emotional or interpersonal situation, and asking to suggest of a few different explanations of this situation OR asking me to ask more. Of course, the prompt needs to have whom I am and similar stuff.
The “Based on…” prompt is simply a horoscope. This is a great piece about how LLMs use the same tricks as psychics to appear helpful, useful, and intelligent.
https://softwarecrisis.dev/letters/llmentalist/
6 replies →
Given how agreeable ChatGPT is built to be this seems like a great way to confirm your own biases. Did it challenge you on your assumptions and viewpoints?
5 replies →
Hmm, interesting. "You might actually be a closet romantic, camouflaged in engineering and anthropology."
I think it's trying to flatter me.
6 replies →
> On multiple occasions, I've gained insights from LLMs (particularly GPT 4.5, which in this regard is leagues ahead of others) within minutes
This is exactly the sort of thing that people falling into the thrall of AI psychosis say.
> For many people—especially those unaware of the numerous limitations and caveats associated with LLM-based models—it can be dangerous on multiple levels.
On what basis do you believe awareness mitigates the danger?