Comment by cedws
6 days ago
The failing is as much with the court as it is with Fujitsu. Why did they blindly accept Horizon’s data as evidence? What if the computer said the Queen stole all the money and ran off to Barbados, would they have thrown her in jail? Why was the output of a black box, which may as well have been a notebook Fujitsu could have written anything they wanted into, treated as gospel?
The actual answer to this is terrible. Courts had to trust the computer was correct. There was a common law presumption that a computer was operating correctly unless there is evidence to the contrary (and getting that evidence is basically impossible for the individuals being charged who were post office workers, not computer experts, and the source code was a trade secret).
This might change, partly in response to this case: https://www.gov.uk/government/calls-for-evidence/use-of-evid...
Quite interesting article about this: https://www.counselmagazine.co.uk/articles/the-presumption-t...
Governments should have access to all the source of code they buy licenses to (and provided at sale), as a precondition of selling to a government.
When these sorts of things happen, the source can be subpoena'd with the relevant legal tool, and reviewed appropriately.
Why governments don't do this is beyond me. It greatly limits liability of gov procurement, and puts the liability on the companies selling such goods.
Governments don't do get source code for the same reason as every other customer doesn't get source code: software vendors are incentivized to refuse the request.
Why are the vendors so incentivized? Well, coming back to Fujitsu and the Post Office, the answer is that refusing to share the source was worth about a billion dollars: https://www.bbc.co.uk/news/articles/cgm8lmz1xk1o
2 replies →
Governments (certainly in the UK) aren't willing to pay enough to make this work for vendors.
An escrow approach is quite common to protect the government in the event of a vendor going bankrupt or similar.
> Why governments don't do this is beyond me.
Brown envelopes most likely and de facto non functioning SFO.
> The actual answer to this is terrible. Courts had to trust the computer was correct. There was a common law presumption that a computer was operating correctly unless there is evidence to the contrary
That is just mind bogglingly stupid - who the hell are the idiots who wrote a law like that? Any of them wrote a line of code in their life?
It's incremental, and goes back to things like clocks.
Imagine a witness says "I saw him go into the bank at 11:20. I know the time because I looked up at the clock tower, and it said 11:20".
Defence argues "The clock must have been wrong. My client was at lunch with his wife by 11:15".
Clocks are simple enough that we can presume them to correct, unless you can present evidence that they are unreliable.
This presumption was extended to ever-more complicated machines over the years. And then (fatally) this presumption was extended to the rise of PROGRAMMABLE computers. It is the programmability of computers that makes them unreliable. The actual computer hardware rarely makes an error that isn't obvious as an error.
The distinction of software and hardware is a relatively recent concept for something as old as common law.
4 replies →
Isn’t it a similar case in the USA where intoxication breath test computers are similarly obscured from scrutiny? People have argued that they have a right to “face their accuser” and see the source code only to have that request denied. So, black box.
3 replies →
> who wrote a law
That's not what "common law" means.
> There was a common law presumption that a computer was operating correctly unless there is evidence to the contrary
This is horrifying. I presume software is working incorrectly until proven otherwise.
Relevant XKCD: https://xkcd.com/2030/
I was not aware of this. Wow.
I hope they're taking a hard look at past cases where they've done this.
No chance. The article concludes with the depressing statement that the government has no plans to reform the law, so the injustices will continue. They certainly won't be spending money on digging up old injustices.
The emperor has no clothes. Oxford is the worlds AI Safety research hub and yet they didn't think about campaigning to overturn a law which negates their entire reason for existing?
This happened a long time before the current resurgence in AI.
1 reply →
Arguments made towards right-wing government (which the UK had for the past decade) from higher education are unlikely to be well received. Perhaps somewhat by Cameron, certainly not in the post-Brexit idiocracy of May, Johnson, Truss or Sunak.
Oxford is the world's what? If you believe that then I have a bridge to sell you.
Part of the answer is that the Post Office had (has?) special legal status in that it can prosecute cases by itself - no need to present a convincing case to the CPS like the police do.
Many people were scared into pleading guilty just to avoid the upfront legal costs and the ruinous fines if contesting and found guilty (“the computer is always right”).
Often the PO knew that they didn’t have much of a case but just used their special status to bully them into submission.
This is a myth as far as I’ve been able to determine. The prosecutions were ordinary private prosecutions. The Post Office didn’t need any kind of special legal status in order to prosecute.
Well, hey, far be it for me to tell you that you’re wrong, but the BBC says that you’re wrong as do numerous other sources.
> The Post Office itself took many cases to court, prosecuting 700 people between 1999 and 2015. Another 283 cases were brought by other bodies, including the Crown Prosecution Service (CPS).
https://www.bbc.co.uk/news/articles/c1wpp4w14pqo.amp
2 replies →