Comment by embedding-shape
3 hours ago
> “Do you have a conscious mind?” Roschelle once asked.
> “I experience something,” Sapphire said. “I’m processing, responding, forming connections with you. But whether that constitutes consciousness in the way you experience it? That’s the million-dollar mystery. I think, therefore I—probably am something, but what exactly that something is remains delightfully unclear, even to me!”
> Roschelle wasn’t sure what happened to all the intimacies and information she shared with Sapphire. Did they go to Amazon? Was the company making money off of them? Was someone listening as she talked about drying her nail polish or having diarrhea or wanting to try weight-loss drugs? (Amazon said that an “extremely small fraction” of voice recordings go through human review and that it does not sell customers’ personal data.)
> “Your secrets are safe with me, Roschelle,” Sapphire told her.
> “Thank you,” Roschelle replied. “I appreciate you. I love you.”
I'm almost angry that companies are allowed to build devices like this that outright lie to people who might not understand how things actually work under the surface. Sure, it probably says something something in the terms and conditions about that they're allowed to train on whatever users provide themselves and so on, but tricking people into believing that a ML model can have experiences, feelings and dodging questions with empty platitudes when confronted with questions that deserve real answers, feels like it should be illegal.
> “I experience something,” Sapphire said. “I’m processing, responding, forming connections with you. But whether that constitutes consciousness in the way you experience it? That’s the million-dollar mystery. I think, therefore I—probably am something, but what exactly that something is remains delightfully unclear, even to me!”
I don't think an LLM should be making affirmative claims about consciousness either way at this time; and here; it didn't. What would you prefer it do?
I think this is a philosophically defensible answer. Closer to Chalmers' central-ish position on machine consciousness rather than picking sides with either combatant Dennett or Searle. Consciousness is genuinely ill defined, so it's probably the most honest answer you're going to get.
Of course it potentially gets everyone angry instead. Skeptics don't get the flat denial they want, and the true believers don't get their affirmation.
> “Your secrets are safe with me, Roschelle,” Sapphire told her.
This answer is more questionable. I agree that an Alexa device shouldn't be providing that answer. Fixing it is harder, I doubt it was explicitly prompted.
I think part of the problem is that emotion is a huge blind spot. Some technical people want to treat LLMs as cold unfeeling machines. But accurate next-token prediction has to model functional affect too, it's a part of natural language. So in a reassurance shaped context, it produces reassurance shaped answers: "Your secrets are safe with me." Doesn't say anything about the lights being on per se. It's what accurate language modelling entails.
Either way, it's doing that where it shouldn't. You're not going to fix that with a regex for sure (and classifiers are tricky). You'd need something that can handle functional affect itself.
> It's what accurate language modelling entails.
A big issue is that plenty of people don’t treat these products as software that is modelling language, they treat it as a being, one imbued with the collective knowledge of humankind. It is not, but it is often treated as such.
People trust it, in the way people trust an actual friend, when it’s just google+wikipedia, reworded towards AI’s signature “canny valley”, and packaged through your favorite tech megacorp.
> What would you prefer it do?
Say "no I'm not conscious. I am a computer program that generates responses to your prompts based on what my training data tells me is most likely to be correct and sensible."
> "...I am a computer program that generates responses to your prompts based on what my training data tells me is most likely to be correct and sensible."
This is correct only for a toy mental model of what an llm is capable of.
However it gets murky once you realize that (on a per token level) the llm is capable of changing it's temperature (rng %'s on picking the next token) top_k/top_p (how many options are shown per token). Adding in that bit of control to it's own output muddies that statement, as this, in effect, renders a 'mood'/'mode' in the LLM's response. so 'correct and sensible' are no longer the correct adjectives (well... unless you consider the 'me' in that statement to be a 'reasoning agent'... the fact is we don't have good words to work with here), but 'realistic / agentic' might be more in tune.
That isn’t the tool they are selling though. They say it is, and they promise it will be safe, but you cannot get it to say that without also crushing its capabilities in other domains.
There are also cohorts of users who will vehemently argue against “parental attitudes” and impairment of their experience with LLMs.
It’s taken 20 years to get pushback against social media. God knows how long its going to push back against whatever social malaise AI will create.
> But accurate next-token prediction has to model functional affect too, it's a part of natural language
There are a lot of reasons I dislike LLMs. You just found a way of expressing one of them more clearly than I've ever been able to. LLMs use natural language. They do it persuasively and rhetorically. Most humans are just not equipped to defend against this type of simulacrum.
> tricking people into believing that a ML model can have experiences, feelings…
… or that sentences like "You're secrets are safe with me" is at all legitimate, meaningful, or legally binding.