Comment by corysama

Sure. Here is the fanfiction book I've been using LLMs to help me write. Helps a lot with improving prose and identifying plot holes. It's much better then a rubber duck for talking out how to improve a chapter and write plausible story arcs. It's not great at word smithing, but I find it errs on the side of too many similies and metaphors, so I just delete some of them as I copy the suggestions over into my draft.

https://github.com/frypatch/The-Price-of-Remembering

I have some science-fiction story ideas I'd love to flesh out. However, it turns out that I'm a terrible writer, despite some practice at it. Also, I can never be surprised by my own writing, or entertained by it in the same way that someone else's writing can.

I've tried taking my vague story ideas, throwing them at an AI, and getting half a chapter out to see how it tracks.

Unfortunately, few if any models can write prose as good as a skilled human author, so I'm still waiting to see if a future model can output customised stories on demand that I'd actually enjoy.

> I do see is genuine concern for the mental health of individuals

I think that is going to be an issue regardless of the model. It will just take time for that person to reset to the new model.

For me the whole thing feels like a culture shock. It was rapid change in tone that came off as being rude.

But if you had that type of conversations from the start it would have been a non-issue.

The place we still call America for illogical reasons is a broken society in seemingly finals stages of its existence. Of course broken people will glom onto yet another digital form of a drug that gives an impression of at least suppressing the pain they feel for reasons they do not understand.

It is little more than the Rat Park Experiment, only in this American version, the researchers think giving more efficient and various ways of delivering morphine water is how you make a rat park.

I don't think chat context and memory count as "hidden features"

> casuals who want to spend as little money as possible to use lots of datacenter time as their artificial buddy

What kind of elitist bs is that?

No this isn't always the case.

Perhaps if somebody were to shut down your favourite online shooter without warning you'd be upset, angry and passionate about it.

Some people like myself fall into this same category, we know its a token generator under the hood, but the duality is it's also entertainment in the shape of something that acts like a close friend.

We can see the distinction, evidently some people don't.

This is no different to other hobbies some people may find odd or geeky - hobby horsing, ham radio, cosplay etc etc.

Where do they all come from? Where do they all belong?

Lack of third-place to exist and make friends.

Wait until you see

https://www.reddit.com/r/MyBoyfriendIsAI/

They are very upset by the gpt5 model

[dead]

This is only going to get worse.

Anyone that remembers the reaction when Sydney from Microsoft or more recently Maya from Sesame losing their respective 'personality' can easily see how product managers are going to have to start paying attention to the emotional impact of changing or shutting down models.

Yeah it’s really bad over there. Like when a website changes its UI and people prefer the older look… except they’re acting like the old look was a personal friend who died.

I think LLMs are amazing technology but we’re in for really weird times as people become attached to these things.

A puppy is just as inhuman as this program. Is it really any crazier to care about one than the other?

Considering how much d-listers can lose their shit over a puppet, I’m not surprised by anything.

>unhinged

It's Reddit, what were you expecting?

Because more than any other phenomenon, LLMs are capable of bypassing natural human trust barriers. We ought to treat their output with significant detachment and objectivity, especially when they give personal advice or offer support. But especially for non-technical users, LLMs leap over the uncanny valley and create conversational attachment with their users.

The conversational capabilities of these models directly engages people's relational wiring and easily fools many people into believing:

(a) the thing on the other end of the chat is thinking/reasoning and is personally invested in the process (not merely autoregressive stochastic content generation / vector path following)

(b) its opinions, thoughts, recommendations, and relational signals are the result of that reasoning, some level of personal investment, and a resulting mental state it has with regard to me, and thus

(c) what it says is personally meaningful on a far higher level than the output of other types of compute (search engines, constraint solving, etc.)

I'm sure any of us can mentally enumerate a lot of the resulting negative effects. Like social media, there's a temptation to replace important relational parts of life with engaging an LLM, as it always responds immediately with something that feels at least somewhat meaningful.

But in my opinion the worst effect is that there's a temptation to turn to LLMs first when life trouble comes, instead of to family/friends/God/etc. I don't mean for help understanding a cancer diagnosis (no problem with that), but for support, understanding, reassurance, personal advice, and hope. In the very worst cases, people have been treating an LLM as a spiritual entity -- not unlike the ancient Oracle of Delphi -- and getting sucked deeply into some kind of spiritual engagement with it, and causing destruction to their real relationships as a result.

A parallel problem is that just like people who know they're taking a placebo pill, even people who are aware of the completely impersonal underpinnings of LLMs can adopt a functional belief in some of the above (a)-(c), even if they really know better. That's the power of verbal conversation, and in my opinion, LLM vendors ought to respect that power far more than they have.

Speaking for myself: the human mind does not seek truth or goodness, it primarily seeks satisfaction. That satisfaction happens in a context, and ever context is at least a little bit different.

The scary part: It is very easy for LLMs to pick up someone's satisfaction context and feed it back to them. That can distort the original satisfaction context, and it may provide improper satisfaction (if a human did this, it might be called "joining a cult" or "emotional abuse" or "co-dependence").

You may also hear this expressed as "wire-heading"

The issue is that people in general are very easy to fool into believing something harmful is helping them. If it was actually useful, it's not an issue. But just because someone believes it's useful doesn't mean it actually is.

Well, because in a worst case scenario, if the pilot of that big airliner decides to do ChatGPT therapy instead of a real one and then suicides while flying, also other people feel the consequences.

Because it's probably not great for one's mental health to pretend a statistical model is ones friend?

Whether the Hippocratic oath, the rules of the APA or any other organization, most all share "do no harm" as a core tenant.

LLMs cannot conform to that rule because they cannot distinguish between good advice and enabling bad behavior.

Fuck.

Not all humans are good friends and therapists. All LLMS are bad friends and therapists.

All humans are not LLMs, why does this constantly get brought up?

Something that used to annoy me about all previous models is that if I asked for a fix to something in a code file (i.e.: fix this method in this class), invariably they would return the entire thing with a bunch of small edits.

GPT 5 is the first model I've used that has consistently done as it is told and returned only the changes.

The great thing about many (not all) "worldbuilding or other creative tasks" is that you could get quite far already using some dice and random tables (or digital equivalents). Even very small local models you can run on a CPU can improve the process enough to be worthwhile and since it is local you know it will remain stable and predictable from day to day.

If you're working on a rented GPU are you still doing local work? Or do you mean literally lending out the hardware?

> the classic 3rd space problem that Starbucks tackled

“Tackled” is misleading. “Leveraged to grow a customer base and then exacerbated to more efficiently monetize the same customer base” would be more accurate.