Comment by rwmj
6 days ago
The actual answer to this is terrible. Courts had to trust the computer was correct. There was a common law presumption that a computer was operating correctly unless there is evidence to the contrary (and getting that evidence is basically impossible for the individuals being charged who were post office workers, not computer experts, and the source code was a trade secret).
This might change, partly in response to this case: https://www.gov.uk/government/calls-for-evidence/use-of-evid...
Quite interesting article about this: https://www.counselmagazine.co.uk/articles/the-presumption-t...
Governments should have access to all the source of code they buy licenses to (and provided at sale), as a precondition of selling to a government.
When these sorts of things happen, the source can be subpoena'd with the relevant legal tool, and reviewed appropriately.
Why governments don't do this is beyond me. It greatly limits liability of gov procurement, and puts the liability on the companies selling such goods.
Governments don't do get source code for the same reason as every other customer doesn't get source code: software vendors are incentivized to refuse the request.
Why are the vendors so incentivized? Well, coming back to Fujitsu and the Post Office, the answer is that refusing to share the source was worth about a billion dollars: https://www.bbc.co.uk/news/articles/cgm8lmz1xk1o
This is why it's unethical for governments to use closed source software. Anything related to government functioning should be auditable.
Then they shouldn't get the contract.
I hope lessons are learned, but I doubt it.
Governments (certainly in the UK) aren't willing to pay enough to make this work for vendors.
An escrow approach is quite common to protect the government in the event of a vendor going bankrupt or similar.
> Why governments don't do this is beyond me.
Brown envelopes most likely and de facto non functioning SFO.
> The actual answer to this is terrible. Courts had to trust the computer was correct. There was a common law presumption that a computer was operating correctly unless there is evidence to the contrary
That is just mind bogglingly stupid - who the hell are the idiots who wrote a law like that? Any of them wrote a line of code in their life?
It's incremental, and goes back to things like clocks.
Imagine a witness says "I saw him go into the bank at 11:20. I know the time because I looked up at the clock tower, and it said 11:20".
Defence argues "The clock must have been wrong. My client was at lunch with his wife by 11:15".
Clocks are simple enough that we can presume them to correct, unless you can present evidence that they are unreliable.
This presumption was extended to ever-more complicated machines over the years. And then (fatally) this presumption was extended to the rise of PROGRAMMABLE computers. It is the programmability of computers that makes them unreliable. The actual computer hardware rarely makes an error that isn't obvious as an error.
The distinction of software and hardware is a relatively recent concept for something as old as common law.
Maybe Napoleon should have conquered Britain after all.
3 replies →
Isn’t it a similar case in the USA where intoxication breath test computers are similarly obscured from scrutiny? People have argued that they have a right to “face their accuser” and see the source code only to have that request denied. So, black box.
Breathalyzers aren't typically considered sufficient evidence in of themselves to convict (or exonerate), iirc many PDs have a policy of treating a breathalyzer hit as probable cause more than anything and then either they throw you in the drunk tank if you don't demand a blood test to verify, or, if they want to actually prosecute you, they get a warrant for a blood test.
AIUI breath test only establishes probable cause. If you fail a breath test you are taken for a blood draw.
Breath test results are routinely challenged (sometimes successfully) by demanding records showing that the device has been tested and calibrated according to the required schedule.
1 reply →
> who wrote a law
That's not what "common law" means.
> There was a common law presumption that a computer was operating correctly unless there is evidence to the contrary
This is horrifying. I presume software is working incorrectly until proven otherwise.
Relevant XKCD: https://xkcd.com/2030/
I was not aware of this. Wow.
I hope they're taking a hard look at past cases where they've done this.
No chance. The article concludes with the depressing statement that the government has no plans to reform the law, so the injustices will continue. They certainly won't be spending money on digging up old injustices.
The emperor has no clothes. Oxford is the worlds AI Safety research hub and yet they didn't think about campaigning to overturn a law which negates their entire reason for existing?
This happened a long time before the current resurgence in AI.
Imagine how much will "machine is right and can't be changed" happen with AI.
Arguments made towards right-wing government (which the UK had for the past decade) from higher education are unlikely to be well received. Perhaps somewhat by Cameron, certainly not in the post-Brexit idiocracy of May, Johnson, Truss or Sunak.
Oxford is the world's what? If you believe that then I have a bridge to sell you.