Comment by jerf

LLMs not only should not be used to replace therapists, if I were to set out to design an AI that would be the worst possible therapist, the LLM may well be the design I chose. It is true that they really aren't "just" "autocomplete on steroids", but at the same time the phrase also does contain some truth to it. They work by extending text in a plausible direction. So if, for example, you have depression, and your word choice and tone indicate depression, you will be prompting the LLM to continue in a depressive direction. You can lead them around, and worst of all, you can lead them around without you ever realizing it.

An example that has hit the news in various forms several times is that if you prompt the AI to write a story about AIs taking over the world, not necessarily by blatently asking for it (though that works too) but by virtue of the type and the tone of the questions you ask, then by golly it'll happily feed you a conversation written as an AI that intends to take over the world. That was already enough to drive a few people halfway over the edge of sanity; now apply the principle to actual mental problems. While they are not generically superhuman they're arguably superhuman in acting on those sort of subtle tone cues, and these are exactly the cues that humans seeking therapy are giving off, but the LLM isn't really "detecting" the cue so much as just acting off of and playing off of them, which is really not what you want. It is way easier for them to amplify a person's problems rather than pulling them out of them. And there have already been other examples of that in the news, too.

I wouldn't say that an AI therapist is impossible. I suspect it could actually be very successful, for suitable definitions of "success". But I will say that one that can be successful at scale will not be just a pure LLM. I think it is very likely to be an LLM attached to other things (something I expect in the decade time scale to be very popular, where LLMs are a component but not the whole thing), but those other things will be critically important to its functioning as a therapist, and will result in a qualitative change in the therapy that can be delivered, not just a quantitative change.