Comment by WhyNotHugo

This exact scenario is described in the 1965 short story "Computers Don't Argue".

You can find it in the following link in the third page of the PDF (labelled as page 84): https://nob.cs.ucdavis.edu/classes/ecs153-2021-02/handouts/c...

It's amazing how 60 years ago somebody anticipated these exact scenarios, yet we didn't take their cautionary tale seriously in the slightest.