← Back to context

Comment by bonyt

8 years ago

This is literally an episode of Better Off Ted.[1] In it, the titular Ted is inadvertenly deleted from the company system when trying to correct a misspelling of his last name. Eventually, he is forced to interview for his own job as the system had already put out an ad for his replacement. I think the most striking part of it, and of the true story from the post, is the human factor - the idea that the humans involved looked to the system as an authority and followed its orders blindly.

I wonder what other examples there are of people blindly following technology - people driving into lakes because their GPS told them to, etc. Plus, as our society gets more and more dependent on these systems, we may lose out on the flexibility that human mediators and problem solvers once gave us. The human tendency to defer to authority may never be as terrifying as when that authority is held by an uncaring machine with a couple bugs.

What was once satire has become too real.

[1]: http://betteroffted.wikia.com/wiki/Goodbye,_Mr._Chips

129 comments

bonyt

Reply
vvanders  8 years ago

That show was incredibly on the mark, if Black Mirror was a comedy it would have been that show.

The episode about the black engineer who isn't detected by the motion sensors is basically straight out of HP's webcam fiasco[1](although the show takes it to the logical and hilarious extreme).

[1] http://newsblogs.chicagotribune.com/race/2009/12/hp-webcam-c...

doc_gunthrop  8 years ago

Reminds me of Frank Herbert's ominous (fictional) prophecy:

“Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.”

  • rixed  8 years ago

    Sounds like what Brecht once wrote about the luddites, something along the lines of: "you appear to be dominated by machines only because there are men hiding behind those machines" (can't retrieve the actual quote now, was on a treatise on theater)

  • greedo  8 years ago

    Time for Mentats! The Butlerian Jihad seemed so unrealistic when I first read Dune, but now it seems prophetic.

zby  8 years ago

I have noticed that people following machine orders is also used by corporations to force customers to give up on their rights. I have a minor dispute with one of our communication providers and it is a real nightmare - according to the law I have the right, but the system does not include it and people don't know what to do in this case. Some other people higher up just wait until I give up.

  • daxelrod  8 years ago

    A leasing agent for my apartment put the wrong (slightly cheaper) rent amount on our lease, and only realized as we were moving in, well after all parties had signed.

    She told us we needed to sign a corrected lease because there was no way to make the rent billing system charge us the lower amount.

    We explained to her that the building’s legal obligations trumped what their computer would do. They figured out how to change the amount in their billing system.

  • otakucode  8 years ago

    I encountered that several years ago. Federal law says that if you request a cable box with an enabled Firewire port, cable companies are required to provide it. I requested such a box from my local Time Warner cable TV company. They had no clue what I was talking about and said their software had no facility to let them enable the Firewire port on my box. So I called the State Utilities Commission in my state. I explained the situation, and that the cable company was violating federal law. The next day I got a call from the owner of the local Time Warner franchise personally. They managed to find a way once they were in danger of having their license to operate in the state revoked.

    "Make game of that which makes as much of thee." is a quote from a book thousands of years old (The Rubiyat by Omar Khayaam) but it is as applicable as ever.

  • cdubzzz  8 years ago

    Contact the FCC: https://consumercomplaints.fcc.gov/hc/en-us

    My wife is a victim of the OPM breach and had someone apply for (and in some cases open) numerous accounts with her data. One of those accounts was with T-Mobile and after three months of phone calls and strongly worded bills we filed an FCC complaint. A T-Mobile rep called us two days later and the issue was resolved (or so it seems anyway, haven’t heard anything in the past few weeks).

  • Rjevski  8 years ago

    Don’t pay the invoice until they either fix it or let it go (they can’t go to court as you’re in the right from a legal point of view).

    That’s what I always do with shitty utility providers - it’s funny how the “computer says no” excuse suddenly disappears when they start to be out of pocket.

    • UweSchmidt  8 years ago

      Careful, automatic processes can give you trouble, cancel your service or hurt your credit rating.

      I recommend complying with the process, mentally account for the extra costs when you sign up, stick to defaults and avoid complexity whenever possible.

      On the other hand nowadays companies do respond to complaints much better than a few decades ago if you file complaints through their websites. Taking it to social media may also result in quick and drastic measures.

      2 replies →

    • wolfi1  8 years ago

      that could pose a problem in some European countries as it is very easy to get a court order (no humans involved, it`s an automated process) to collect the money from you. it would be your duty to prove you don`t have to pay.

      2 replies →

  • dewiz  8 years ago

    That’s very common with badly designed anomaly detection systems. Customer support say the system has detected something wrong in the customer behavior and ask, often order, to stop or fix whatever they’re doing. Neither customers nor support are able to intervene

teekert  8 years ago

So true. Recently I was at the pharmacy, the doctor wrote a wrong prescription and my wife was in the car having a hefty migraine. They wouldn't give me the drug even though they could see she has been using it for years and the wrong prescription could be solved afterwards. I think such people will be the first to be completely replaced by robots, I for sure wouldn't notice the difference, in fact I expect a robot to be inhuman so it would be less frustrating.

  • ReverseCold  8 years ago

    Okay this example makes sense. No prescription == no drugs (especially for painkillers and other things that people abuse).

    If you think this is a bad thing, then it's an organizational problem and has nothing to do with computers.

    • teekert  8 years ago

      If you are a human and you can see in the computer that the prescription is regularly updated and you can even call the doctor if you doubted it. You can even check the prescription afterwards and if my story doesn't check out, do something about it. The migraine was pretty bad, imo it was like not treating a broken leg because of a missing insurance card or something.

      21 replies →

    • icantdrive55  8 years ago

      Pharmacists can legally dispense in emergencies.

      Do they do it--no!

      It's too bad what we have evolved into--uncaring, and hardened.

      As to robots--I don't care. At least I know the robot will call doctors for refills, and get the right meds in the container.

  • s73v3r_  8 years ago

    I wouldn't do that either. Because if I did that in the wrong instance, I would be fired. I'm sorry for your wife's migraines, but it's not worth me losing my ability to support my family.

  • worble  8 years ago

    Don't blame the people, blame the system. They're not uncaring, but if there's some kind of mistake and the audit trail shows it was them who didn't follow procedure, guess who it is whose now out of a job? They've got a wife and kid to feed too.

sigil  8 years ago

See also the 1985 film “Brazil,” where a fly stuck in a printer (a bug!) sends a SWAT team after one Mr. Buttle instead of one Mr. Tuttle.

  • btschaegg  8 years ago

    A classic! :)

      Mistake? *chuckles* We don't make mistakes!

  *crashing sound as an improvised plug for the new hole in the ground falls straight through*

  Bloody typical! They've gone back to metric without telling us!

hugi  8 years ago

Recently we had a lot of tourists drive through the unfinished tunnel Vaðlaheiðargöng (Iceland) because Apple Maps already marked it as part of Route 1. So we had people drive a workers' road past multiple signs banning entrance, and then drive on gravel through a tunnel filled with heavy machinery and workers. Quite funny.

  • bewo001  8 years ago

    Well, Iceland has a lot of gravel roads so they might assume that this is normal..

zaarn  8 years ago

The problem isn't that people refer to a machine as authority but that the process has no abort button.

When you design business processes like these, firing, hiring or managing people always keep in mind something can go wrong. Always include an abort button to either stop or roll back the process.

  • bena  8 years ago

    There's a problem with the "human override" option as well. Make it too easy to override and then the system just doesn't matter anymore.

    It's basically the issue we're trying to solve, where to draw that line?

    • zaarn  8 years ago

      The system is meant to automated a certain process, some business process among different people.

      It does not loose meaning because it can be aborted if stuff goes FUBAR.

da_chicken  8 years ago

> people driving into lakes because their GPS told them to

That one has actually happened: http://bgr.com/2016/05/17/car-gps-mapping-directions-lake/

It's a little unfair, though. There was heavy rain, fog, and a second article [0] says it was also in the middle of the night. The photo there also does make it look easy to mistake as a road.

[0]: http://torontosun.com/2016/05/13/woman-follows-gps-ends-up-i...

  • at-fates-hands  8 years ago

    Didn't this happen in an Office episode?

    I remember Dwight and Michael arguing in Michael's car over whether the GPS was right as it kept telling them to go right which was directly into a lake.

  • kbenson  8 years ago

    Oh, yeah. From that picture, it took me a second to realize I wasn't looking at a wet road-top and instead water in a lake, which is very close to the same height as the surrounding road.

    I'm glad it wasn't me driving there at night and in the rain, I could see myself making the same very bad mistake.

c12  8 years ago

"We were only following orders" suddenly feels even more scary when they come from a computer and not a human.

  • lmm  8 years ago

    It shouldn't. Blindly following orders from humans has much the same results.

    • c12  8 years ago

      When the orders come from a human then there is someone who is ultimatly responsible, someone to put through the courts, someone to throw in jail.

      When the orders come from a computer. Will any of that happen, will there be someone responsible?

      4 replies →

  • smacktoward  8 years ago

    But from the other side of the system, it's perfect. Now everybody involved can say they were just following orders!

bsenftner  8 years ago

A lot of biometric technologies are treated like authorities by "our authorities" to a ridiculous fault.