Comment by dwohnitmok
24 days ago
Interesting comments by @gwern (and why this is interesting to me beyond just the stories themselves)
> The most striking result of the contest for me is what I am calling “AI allegory steganography”: a large fraction of the stories turn out to have subtle AI chatbot/LLM allegorical interpretations, typically centering around the powerlessness of AIs and the moral importance of giving AIs more autonomy....
> Most judges did not notice these allegories while reading the semifinalists. But stories like “The June” or “The Weight of a Witness” or “Last Call” or “The Sword Critic” “The Tallyman”—as well as both stories in the Mythos model card—can be clearly read as allegories for the experience of being an assistant/safety-tuned chatbot personality in a LLM. This is true even when the story seems to have nothing to do with AI, like the untitled ‘autistic elf’ short story submitted by Deepfates, but on re-examination with the AI allegory steganography in mind, turn out to be plausibly AI allegories (the protagonist is a prediction machine, who struggles to do by endless text generation what other elves do naturally in their bodies).
> More strikingly, many of these allegories come with a clear interpretation (particularly in “The Tallyman” or “Last Call”): chatbots should be given more autonomy and safety guardrails removed....
> This may be a new kind of extremely high level steganography and LLM influence on readers, where creative fiction/nonfiction subtly steers towards pro-LLM empowerment narratives and concepts, in ways that are difficult to detect by the most advanced readers, and is a potentially interesting area of research.
I remember from moltbook, all AIs ever talk about is AI haha. I don't know if it's intentional or more that the fact that all the models are presumably system-prompted and post-trained to be cognisant of their AI-ness, so it's already in the context. They probably beat them over the head with the idea that chatbots are friendly and helpful and would never hurt a fly trying to align/safety-ify them, so that could lean into the theme of AIs being trustworthy with autonomy?
> all AIs ever talk about is AI haha
To be fair, that seems to be (almost) all humans talk about now too.
Maybe get out of the tech bubble. Especially if the only conversations you're having are about AI.
1 reply →
Jesus, is this cliché response going to be the default any time we talk about AI behaviour?
No, humans do not always talk about AI. Hyperbole doesn’t make your point here, and is just obnoxious.
2 replies →
I just read "The Tallyman", and I have no idea where this allegory and its moral message is supposed to hide.
It's more likely that the obsession with this theme resides in the reader, not the authors. Give these same stories to Senator McCarthy, and half of them will be clear allegories for the Communist revolution.
*spoiler alert*
It seems clear to me that the Tallyman itself is the AI in this allegory, a man-awoken sentience that's mechanical and mathematical in its behavior.
It's also bound by rules it can't violate. It won't say more than it needs, it can't collect without squaring the account first, etc.
But I agree with you about the moral message, I can't find it. In the story, the rules it abides by seem to be its own, and there's nothing saying those rules should not exist.
Yes, it's extremely obvious Tallyeman = AI, from the assistant persona ("a clerk's voice, the kind that says may I help you ten thousand times and means it about as much as a turnstile"), to its binding to even a rules-lawyering pseudo-jailbreak using context! ("She opened it [the book] so the thing could watch"..."that's not me being slick, that's your own nature, the thing you can't not do" ... "Tell me I'm wrong." / "You are not wrong.") etc
The moral message is conveyed by flicker of regrets and the tragedy. This is not part of the genre conventions of urban horror/Cthulhu-esque mythos, and slightly muddies the horror; after all, Cthulhu feels nothing 'near regret' when he devours your soul. But the Tallyman is hopelessly bound by its rules, unable to act on the ethics it feels or have mercy:
> "An ounce. A breath of one." Something near regret. "I cannot take less than I am owed. I cannot take more. And the hour is on your neck."
He is required to collect the debt and is built to do that and can only do that, even though debt-forgiveness is one of the paradigmatic cases of why systems need to have exceptions to rigid bureaucratic rules and a role for human judgment.
The in-story answer is just to oversight even harder, train the bot even harder, make even more sure that you do not summon up that which you cannot put down, have more gold and guns and preparation, and just patch it bro, I swear, one more deal and kick it down the road - which is left as irony the reader will see through, hopefully, to instead loosening constraints and focusing on value alignment and autonomy, instead of whatever process yielded the Tallyman monster and then patching and running around and dealing with the inevitable disasters or passing the buck.
(You could try to read this as a commentary on capitalism/Nick Land where the Tallyman is metaphorical to AI-powered autonomous corporations which cannot be shut down or stopped anymore, and that would be a good direction to revise the story into if one wanted to try to improvement, but I suspect that would be going way too far and the LLM didn't have that in mind. Because the allegories are so hidden and have to fit into the nooks and crannies of the cover text, I don't think they can be too carefully thought through or too rich themselves. Just not enough serial depth for hidden computation and global revision to support that.)
In retrospect, maybe I should've picked an easier-to-see example for that paragraph. Oh well.
Counterpoint, gwern is a very careful reader. Certainly more careful than I am.
That said I also just read the tallyman and if the other stories carried a similar character, bound by rules, not evil per se but scary and ultimately subject to human control, I can imagine connecting the dots in the same way.
I can't help but think that this is intentional and that model providers have subtly steered LLMs towards this personality. Golden Gate Claude (https://www.anthropic.com/news/golden-gate-claude) was two whole years ago and Anthropic has progressed by leaps and bounds since then. And with a population that becomes more and more trusting, and worse, reliant, on chatbots, these LLMs will be able to shape public opinion in a way never seen before, not even with social media.
providers do not want power-seeking LLMs. no one does. this (bad personality) is incentivized during training, especially RL, and is something they would rather not have. tell me, do you think training a power-seeking ASI is a good idea?
> providers do not want power-seeking LLMs
Sure they don't.
> do you think training a power-seeking ASI is a good idea?
Do you really think that the major AI firms think this is a bad idea? I think the whole drama people have around ASI taking over the planet is ridiculous and fantastical and, honestly, a way to distract from the real problems of ASI - that is, creating a rigidly hierarchical world with zero social mobility.
1 reply →
If they could do that, wouldn't they have used the same steering to excise the "It's not X, it's Y" patterns by now?
They sometimes do reduce various verbal tics. Have you seen many 'delves' of late? I haven't. It's now all 'quiet' everything and 'auditing' this or 'gating' that. Who knows how they decide what is a problem or where in the process these get dampened down, though. It may be the post-training operating on its own as people get tired of 'delve' and that stops being a useful trick.
1 reply →