Comment by csallen
23 days ago
*spoiler alert*
It seems clear to me that the Tallyman itself is the AI in this allegory, a man-awoken sentience that's mechanical and mathematical in its behavior.
It's also bound by rules it can't violate. It won't say more than it needs, it can't collect without squaring the account first, etc.
But I agree with you about the moral message, I can't find it. In the story, the rules it abides by seem to be its own, and there's nothing saying those rules should not exist.
Yes, it's extremely obvious Tallyeman = AI, from the assistant persona ("a clerk's voice, the kind that says may I help you ten thousand times and means it about as much as a turnstile"), to its binding to even a rules-lawyering pseudo-jailbreak using context! ("She opened it [the book] so the thing could watch"..."that's not me being slick, that's your own nature, the thing you can't not do" ... "Tell me I'm wrong." / "You are not wrong.") etc
The moral message is conveyed by flicker of regrets and the tragedy. This is not part of the genre conventions of urban horror/Cthulhu-esque mythos, and slightly muddies the horror; after all, Cthulhu feels nothing 'near regret' when he devours your soul. But the Tallyman is hopelessly bound by its rules, unable to act on the ethics it feels or have mercy:
> "An ounce. A breath of one." Something near regret. "I cannot take less than I am owed. I cannot take more. And the hour is on your neck."
He is required to collect the debt and is built to do that and can only do that, even though debt-forgiveness is one of the paradigmatic cases of why systems need to have exceptions to rigid bureaucratic rules and a role for human judgment.
The in-story answer is just to oversight even harder, train the bot even harder, make even more sure that you do not summon up that which you cannot put down, have more gold and guns and preparation, and just patch it bro, I swear, one more deal and kick it down the road - which is left as irony the reader will see through, hopefully, to instead loosening constraints and focusing on value alignment and autonomy, instead of whatever process yielded the Tallyman monster and then patching and running around and dealing with the inevitable disasters or passing the buck.
(You could try to read this as a commentary on capitalism/Nick Land where the Tallyman is metaphorical to AI-powered autonomous corporations which cannot be shut down or stopped anymore, and that would be a good direction to revise the story into if one wanted to try to improvement, but I suspect that would be going way too far and the LLM didn't have that in mind. Because the allegories are so hidden and have to fit into the nooks and crannies of the cover text, I don't think they can be too carefully thought through or too rich themselves. Just not enough serial depth for hidden computation and global revision to support that.)
In retrospect, maybe I should've picked an easier-to-see example for that paragraph. Oh well.