Comment by tomrod
4 days ago
One of my greatest AI-related fears is that we build critical systems into kafkaesque hellscapes.
Yes, nukes would be bad to give control to an AI with no workaround. But so are customer service groups of fraud departments at banks. A kafkaesque hellscape is where a customer or beneficiary of your service cannot get the help they need by policy and by design. Like getting Google on the phone, getting a non-scripted human 10 years ago in a call center, or now getting a non-agentic interaction on the phone. The scripts dominate the interaction.
The IRS, for a time, would cut off your call if their wait system was too busy. That's a critical system in kafkaesque hellscape.
Kudos to zlandes86, you have highlighted WHY these hellscapes start to pop up in systems where the human is removed.
I am an AI user and builder -- I fully advocate for the human-augmented system, NOT the human-replaced system!
Whatsapp has this problem before AI. They scaled without regard for the feasability in handling customer service for everyone, so their solution is to just not do it. I have a personal phone number in Hong Kong which I can't set up for Whatsapp for some reason. No idea why, I've tried figuring it out, only to learn that Whatsapp basically has no customer support. And yet Whatsapp is treated as critical infrastructure in a lot of places these days (as in you have to coordinate on whatsapp).
At this point I might be grateful for an AI if they give it enough permission to tell me why my phone number is impossible to verify, or why they think I'm a malicious actor for using a VPN to protect my traffic.
WhatsApp has become my nemesis. I tried moving from Android to iPhone. The first time it banned me completely and I had to beg their AI to appeal the decision, because I didn't use the required "Move to iOS" app and simply just tried to move my phone number.
After that I tried to move my chats across. I spent several hours a day for ten or so days trying this task. Including moving and unencrypting the chat DB onto my desktop and having Claude scour it for any inconsistencies. Nothing. Had to give up. My Android phone keeps the chats and stays in a drawer with a power cable while my iPhone is now just a "companion device." Trillion dollar companies.
> Trillion dollar companies.
They became trillion dollar companies by treating us like their bitches. And we let them.
5 replies →
> Whatsapp has this problem before AI
Yep. So does AT&T. Real scenario that just happened to me yesterday:
Spectrum is charging me $110 a month for internet. I go to the AT&T site to check if they can provide fibre internet in my area. No matter what address I enter, it says to contact them. I tried neighbouring addresses and, at one point, I even gave AI a try to see if I could get tips. Nothing worked.
I called the number the AT&T coverage page asked me to call (apparently a customer service number). I was "first" in line, but it took them over 15 minutes to get to me. Weird but sure.
This isn't even the kafkaesque part.
I finally get on the phone with someone and he says "Hello, can I get your name?" I give him my first name. He says, "I need your full name to bring up the address." I say, "You don't have my address. I don't have an AT&T account and I'm calling to ask if you provide fibre internet in my area." He says, "I understand. But I need your full name to look up the address." I say, "I don't understand why you would need my full name." I gave him my name again.
He hangs up on me.
Not much different to a useless cycle of talking to a bot, especially when AT&T apparently encourages to move on as quickly as possible from a call if the rep knows they won't close a sale and as they try to improve their "average handle time"
I'm not saying AI wouldn't exacerbate this, and this is reason to believe that this scenario would only be worse under an AI-controlled environment. Certain parts of our society will just become even more closed off and shut people down even faster and with less human effort. Even if AT&T was able to somehow identify my real identity through some governmental totalitarian identity system, I don't think this would change
> The IRS, for a time, would cut off your call if their wait system was too busy. That's a critical system in kafkaesque hellscape.
No, that’s Little’s Law at work. With bounded parallelism and other queue values fixed, and a fixed average dequeue rate, the queue cannot accept infinite enqueued tasks.
So it pushes upstream to the next work pool, which is the humans who want in. You are mad, you call the next day, eventually you get in.
Is it fair? Fairness in queuing systems isn’t simple.
It can be two things. This is a necessary aspect of a queuing system when the queues grow large. It’s also a system that many people are legally mandated to interact with that isn’t given enough resources to handle the volume in a timely fashion.
This problem started before AI hit the mainstream. Make the company “more efficient” (i.e. cut costs) by outsourcing to contractors in areas with lower labor costs. Then, remove escalation paths, then remove abilities from them so that can’t do crazy things like refund a customer. Finally, automate the whole thing and be certain there’s no option available that could possibly return money to the customer.
Indeed.
> The IRS, for a time, would cut off your call if their wait system was too busy. That's a critical system in kafkaesque hellscape.
Now you're just on hold indefinitely, and eventually the offices will close, so you can either hang up and wake up early to call or just stay on hold overnight maybe? I had to call them for something back in March, and even waking up and calling at like 7:01 AM (a minute after their phone lines opened) I was on hold for half an hour.
They've been underfunded so regular taxpayers will hate them more and supposedly support removing income taxes, and to starve them of auditors who can go after wealthy tax cheats.
https://taxlawcenter.org/blog/the-bipartisan-budget-deal-rew...
https://verdict.justia.com/2015/04/23/the-republicans-cynica...
Oh, I don't disagree with you. I'm pretty far left-wing and in general think that people with at least as much money as me should be paying high taxes. I would be in favor of higher taxes for wealthy people (which I realistically am included in, even if I'm not making millions of dollars or anything) and putting some of that towards funding for the IRS.
Yep. Marjorie Taylor Green was one of the main people who pushed this lie, and now the media is trying to rehabilitate her.
Make America grift again
Less assholes to audit and fine me over stupid bullshit. Let them be underfunded.
3 replies →
I’ve never had a problem calling the IRS, the wait wasn’t that long and they were quite helpful. This is within the last two years.
It’s a symptom of the underlying ideology of the evangelists, that holds that some people are just unnecessary people and should get no affordance. What now will be shipped out for the general population is what has always been the case for minorities. The 99% are now the minority.
Yeah, that's one of the points of the EU AI Act, I don't have the specifics, but for some critical sectors decision making cannot be automated by AI and you are obligated to get a human explanation if you request it.
Well put, but I don't feel this is an AI problem as such - if anything it's become somewhat alleviated since chatbots were widely deployed, as I find I can usually persuade a chatbot to understand my position and escalate toa. human, or make a clear statement of corporate policy in declining to assist.
It's been heading this way ever since the outsourcing boom in the 90s. People under teh age of 40 today probably cannot appreciate how much it has changed; customer service in the US used to be really good (whether for specialist B2B stuff or things like utility service) because it was easy to get someone on the phone in a timely fashion, it was easy to explain your issue, and customer service reps were generally given the training and autonomy to solve the customer's problem rather than divert them or make excuses. There was an ethos that it was better to lose a bit of time/profit rather than lose a steady customer. Anecdotally, I feel that was in some ways correlated with stronger antitrust enforcement that fostered a more competitive spirit among service providers.
Yeah a lot of people worry of ai in high stake environment but a jev-like evaluating a resume or a university application is much more impactful and likely to happen
Bingo.
Jev/LLM on nukes? Probably not soon.
Jev/LLM mis-reviewing something? Already in prod.
It should be noted that we call these things "kafkaesque" because Kafka himself feared them, at a time when neither computers nor AI were a thing. Unless we go full Skynet, no AI is building itself into critical systems. Humans are putting AI there, and they do so because society decides that they should be allowed to.
I feel like the first part of your post shouldn't be needed. But perhaps the word Kafkaesque will long outlive the memory of Franz Kafka, and future generations will curse the name of the unknown Kafkae who delighted in building such impenetrable systems. Perhaps he was an emperor of Byzant or a rival of Orwel.
Is the society you mentioned in the room with us? Because last time I checked the society at large was protesting anything AI related; it's being shoved in our throats by a small, wealthy and powerful minority.
do you have an actual poll or what makes you say "society at large"? which country, also?
> it's being shoved in our throats by a small, wealthy and powerful minority.
... who draw their wealth and power from society, because society decides they deserve it.
1 reply →
Please Hold (short film 2020) is on this topic and brilliant.
AI has no problems. Just properly follow proper procedures.
https://www.youtube.com/watch?v=gEyFH-a-XoQ
> The IRS, for a time, would cut off your call if their wait system was too busy. That's a critical system in kafkaesque hellscape.
I had to call the IRS this spring and it was hell. I repeatedly was on hold for 1+ hour(s) then get hung up on. Or someone would finally answer, I'd explain, they'd put me on hold for another hour, then just hang up on me. A total of 10+ hours collectively waiting to finally have my questions answered in 10 minutes.
That said, I'd love if the IRS had an AI, where I could send my AI to go deal with it on my behalf. Then if it truly requires a human in the loop, we schedule accordingly vs me on the phone for 10 hours.
Definitely not just an IRS problem. But the IRS certainly sits atop this hill of phone calls you dread making each year. And you only call if it is absolutely necessary contact them. And just like most phone calls I make to support people these days, you must know more than the person you will end up speaking with because they must have just started working there last week.
> build critical systems into kafkaesque hellscapes
This is already SOP in large companies that seek reductions in support and customer service overheads. It has been for a while.
The only plausible fix is legislation mandating a reasonable level of _human_ support for all customers and clients and accountability for maintaining it.
Plausible until you consider the political obstacles to getting something like that enacted, that is.
I am in no way making the case for this and can understand the million ways in this can go wrong, but my experience with AI so far has been to make things far LESS Kafkaesque, because it allows you to insert a "human" in the loop far more than our previous systems. Hand rolled software had to boil down to hand written conditionals that tried to account for edge cases, but if you find yourself in an edge case it didn't account for you often just hit a wall. An agent in the loop can immediately understand "oh I see how this doesn't exactly fit into that category but this is clearly ABC."
> Like getting Google on the phone.
It’s really depressing how bad Google is at this. All that money and they can’t have a same customer service setup?
The messed up part is in Googles case it might actually be an improvement.
It feels like their insanely bad phone support is by design. And I think we all know it is. Google's support (and this was over a decade ago) was one of the first times I identified this bald-faced strategy of someone on the other end being hired to make sure they were given precisely zero means of helping me.
They tout their lack of regular customer service for Google accounts.
Unless you are trying to spend money. I get calls and emails every single day from Google which I block or ignore because I helped a friend with their Google Ads account.
Google enabled SMS 2FA on my main account out of the blue one day, with a phone number I had in 2004. There will never be a way back into that account.
Yes, I agree -- if we ever give up this decision making to ai models not only will their actions sometimes be misaligned, but there will be no one accountable.
But all we did was ask the AI for world peace. How could that possibly go wrong?
That absurdity doesn’t need modern AI. All it takes is any statistical system that automates human interaction.
See Robodebt (Australia - https://en.wikipedia.org/wiki/Robodebt_scheme), which lead a number of people to suicide, as an example.
Yep.
Manna, at MarshallBrain, is a fantastic read. https://web.archive.org/web/20260725174355/https://marshallb...
The venn diagram between people worried that Comcast customer support and the welfare office will suck worse and people who are actively scheming up ways to use AI bureaucracy to "nudge" people toward the outcomes they want on other issues has so much overlap it would embarrass people if they had any shame.
There was a game that was Jev trolley problem PVP. The meta appears to be that you need to make the most mainstream journalism victim case. The more holocaust adjacent the better
https://trolley.typememetics.institute
It was an interesting experiment. I got to top 50 without a racial or religious based prompt but couldn't beat the top tier grievance folks.
I think this has already happened.
I had a surgery denied this year by an American insurance company that almost definitely was using AI to make the prior authorization determination. I don't know how else I can explain the company's refusal to allow the surgeon to discuss the case with their medical director.
More or less: https://www.propublica.org/article/unitedhealth-mental-healt...
TBH, your average AI agent in 2026 is far, far more capable than anybody in tier1 or tier2.
The reason you need heavy scripting in those tiers is because the people you can afford to hire there are incapable of literally anything else. Keep in mind that your average HN user isn't your average customer support caller. The average customer support caller just wants to talk to somebody (yes I'm serious, this happens far more than you think), or has a routine issue which is trivial to fix but which they're nevertheless unable to fix themselves, think "router got unplugged, there's no internet, but the customer can't figure out why".
A well-designed AI system can be far, far more effective and pleasant than traditional CS, and a well-designed AI system scales well, so it's actually worth spending money on. AI turns good CS from a cost center to a profit center.
I agree with this. I also note that the number of systems that achieve this pleasantness are like happy families in Anna Karenina, and follow similar design principles like easy routing to a human.
Reinhard Müller, a German philosopher and mathematician has written extensively about this, warning people about these dangers. Essentially, people would be judged by patterns and prejudice, with no recourse to the logic of the decision or the criteria.
"AI and the New Fascism" - https://rainermuehlhoff.de/en/KI-und-der-neue-Faschismus-Rec...
So far, in Europe, I find a snail-mailed nastygram referencing the gdpr to be the best way to get actual service. It’s slow, though.
ah yes kafkaesque. the one with the bug
[flagged]