Comment by jmyeet | Hacker News Reader

Comment by jmyeet | Hacker News Reader

If I go into a police station and file a police report saying you stole $50,000 from the Post Office branch you ran, one would hope that I'd also have to prove it. If I filed a civil complaint against you, I'd have to produce evidence.

In both cases, if I lied by, for example, saying that I had evidence that I did not, I could and should be charged with a crime, in the very least making a false report and/or making a false statement to the court.

Yet we let companies do this with IT systems all the time. Hertz famously accused people of stealing cars that they did not [1]. The same principle should apply: whoever OKed such a system should be charged with a crime.

All these systems make the same mistake: they want to replace humans. If you want to do that, you should be on the hook legally for the results but really it shouldn't be allowed.

It's completely fine if Hertz designed a system that flagged people for not returning cars (ie stealing them). That would get sent to a person. That person would review the information, check if it was an error or not and, if not, file a police report. That human would be responsible for maaking a false police report in the case of an error. Any errors could then be addressed as bugs with the systems provider.

Likewise, for Horizon, by all means automate the review of financial records. Find what you think are fradulent transactions. That gets spit out to a report. That report may then be investigated by, say, a financial accounting firm who is responsible for determining the facts and, if necessary, taking legal action.

You see how this version aids a person doing this job and the actual versions of these replaces the humans? A human can't necessarily review every transaction of the postal system so automating fraud detection makes sense.

What doesn't make sense is allowing the systems provider and the government agencies that enabled this off the hook because of "bugs".

[1]: https://www.npr.org/2022/12/06/1140998674/hertz-false-accusa...

Not to excuse the post office or the UK legal system-- but there are reasons your point has limitations.

The subpostmasters were signing statements that their accounts were accurate, their charges were mostly related to these sorts of collateral lies rather than the theft. So in a uselessly pedantic and literal sense they often had performed the wrongful act they were accused of, though this ignores that they were more or less forced into it and sometimes explicitly told by tech support to do so.

In specific cases the financially disabled and often unrepresented subpostmasters made legal errors. For example, instead of defending themselves by responding that the Horizon figures were inaccurate they defended themselves by saying they could prove they didn't take the funds from their own books. Essentially allowing the burden of proof to be reversed.

None of that is incompatible with what my comment.

So subpostmasters signed affidavits to the accuracy of their accounts. The burden of proof should still be on the government to substantiate that those claims are false and there is material harm and not, for example, just a bookkeeping error.

Particularly when a system is new, we all know that it has to be verified. For migrations that typically means comparing the results to the old system and then manually looking at discrepancies. In the case of a new system, it should be the bare minimum standard that before a person is accused of a crime, the system is verified, in this case by human review, probably by an auditor or financial accounting firm.

The crime being the false affidavit (even here you have the government has the burden of proof to show it was knowingly false) rather than any alleged theft is pure theater and a cop out. Like you sign a statement saying you didn't steal money and I charge you with lying. Shouldn't I have to prove you were lying? Isn't that basically the same as proving the theft occurred?

If subpostmasters can be charged and/or sued for making false statements by a computer system, why can't someone be charged for the false statements that computer system made?

No, you've not understood what happened.

The SPMs were forced to sign and take responsibility for accounts that were prepared by the Horizon system, which claimed the SPMs owed the Post Office tens of thousands of pounds of imaginary money. If they later claimed that the accounts were wrong (which they were), they'd be prosecuted for signing them, and if they didn't do that they would be prosecuted for not paying the PO the money they supposedly owed. So they would lose either way, and the actual accuracy of the accounts was never tested in court, because in any individual case the PO did not disagree with the SPM about that.

The point being that this can't be blamed on software bugs alone. It was caused by different departments of the PO having completely contradictory ideas about how the software was supposed to work.

I think you have it backwards, they signed statements saying the system generated false accounts were accurate. In some cases they did so after putting in manual adjustments to diminish the debts which themselves were not accurate (they had the right moral effect, but they were literally false accounting records-- correct accounting would have required finding and adjusting the actual errors made by Horizon, which they did not and likely could not do).

At best they were in a weird catch-22 where disputing the records is an admission that they knowingly filed false accounts (itself a crime, regardless of they funds owed or not), and at worst they indisputably entered false accounting records (though for understandable reasons and sometimes at the direction of tech support staff).

Part of the reason justice was possible at all was because Alan Bates simply refused to sign the false reports. He got shut down and suffered tremendous hardship as a result, but he wasn't litigated against-- which left him in a position to drive forward justice. ... and even that required whistle blowers and leaks to force the disclosure of documents that were wrongfully being concealed.

I can't emphasize enough that I'm not saying any of this to attribute fault to the subpostmasters or deflect it away from the post office or Fujitsu. I don't think it does.

But I think it's important to have a more comprehensive understanding of what went wrong so as to understand how situations like this can be prevented.

To that end understanding how the situation made some of the subpostmasters guilty (or at least in violation of their contract) in narrow a technical sense is important, because it highlights how the judiciary and its duty bound actors (the solicitors, barristers, and independent experts) operated in a -- for lack a better word-- autistic and rule bound way that yielded results which were completely incompatible with justice.

>Any errors could then be addressed as bugs with the systems provider.

Were bugs actually involved? I could only find this sentence in the article and it leaves it open if it was a bug or an user error.

>Many of the Hertz cases involved customers who had called to extend their rental agreement, but the extensions were not properly reflected in Hertz's computer systems.

There are hundreds of cases, all different. Many involve software bugs but at least one does not. You can read about that one in the common issues judgement paragraph 217.

My comment was about the Hertz car rental system, not Horizon.

Regarding software bugs in Horizon: Some of the bugs are mentioned in the early chapters in the book by Nick Wallis. I first read an article by the Guardian which only mentioned two of the bugs (as far as I remember) but also said many other bugs were discovered [1]. I first thought these were mostly UX issues [2] [3] that caused user input errors, but reading some of the inquiry reports and the book made it more apparent that there were system bugs, e.g. [4] on top of a sub-optimal message system [5]. Of course, you can say UX issues are also some kind of bug, error or defect.

[1] >As early as 2001, McDonnell’s team had found “hundreds” of bugs. A full list has never been produced, but successive vindications of post office operators have revealed the sort of problems that arose.

[2] >“The Dalmellington Bug entailed a user repeatedly hitting a key when the system froze as she was trying to acknowledge receipt of a consignment of £8,000 in cash. Unknown to her each time she struck the key she accepted responsibility for a further £8,000. The bug created a discrepancy of £24,000 for which she was held responsible.”

[3] >"If a bunch of products sit in a basket for long enough on the screen Horizon will turn them into a sale automatically." -- ‘It is an example of Horizon doing what it was supposed to do.’ -- ‘It is evidence of the system doing something without the user choosing to do it.’ retorts Mr Coyne. >"This stance of blaming the users if they were confused by Horizon’s design was taken to an extreme with “phantom transactions”. These were transactions generated by the system but which were recorded as if they had been made by a user"

[4] >In April 2003, Horizon version S30 was released by Fujitsu. It had not been properly tested, and it introduced a bug known as the Reversal bug. If a Subpostmaster inputted a series of transactions, and then for some reason wanted to cancel them, the transactions would have to be reversed out of Horizon. The Reversal bug appeared to reverse a transaction, but in fact doubled it. This was due to a plus sign in the code which should have been a minus sign. The Reversal bug was discovered by Fujitsu after Subpostmasters began to complain about the growing amount of non-existent money Horizon calculated as being in their accounts at the end of the week. When one Subpostmaster reversed an internal £13,910 cash transfer out of one stock unit and into another, he managed to create a £27,820 negative discrepancy in his accounts.”

[5] > "The biggest problem, however, was the information being written to something called a message store.” >“According to Clint, Horizon’s developers had spent the last couple of years ‘firing any old shit’ into the message store as they changed and developed the product. Clint explained that this was a recipe for disaster. If a message was given a new data field, but this wasn’t retrospectively applied, the counter software could retrieve an old message with what it would now perceive to be a missing data field. The reaction of the application would be unpredictable.”

Reply on news.ycombinator.com