Resolving disputes through computer evidence: lessons from the Post Office Trial

On Monday, the final judgement in the Post Office trial was handed down, finding in favour of the claimants on all counts. The outcome will be of particular interest to the group of 587 claimants who brought the case against Post Office Limited, but the judgement also illustrates problems handling evidence generated by computers that have much broader applicability. I think this trial demonstrates that the way such disputes are resolved is not fit for purpose and that changes are needed in both in how computers generate evidence and how such evidence is reasoned about in litigation.

This case centres around disputes between Post Office Limited and sub-postmasters who operate Post Office branches on its behalf. Post Office Limited supplies these sub-postmasters with products to sell, and the computer accounting system – Horizon – for managing the branch. The claimants contend that shortfalls between the money that was in their branch and what Horizon says result from bugs in Horizon or someone maliciously accessing it. The Post Office instead claims that the shortfalls are real, and it is the responsibility of the sub-postmaster to reimburse the Post Office.

Such disputes have resulted in sub-postmasters being bankrupted, and others have even been jailed because the Post Office contends that evidence produced by Horizon demonstrates fraud by the sub-postmaster. The judgement vindicates the sub-postmasters, concluding that Horizon “was not remotely robust”.

This trial is actually the second in this case, with the prior one also finding in favour of the sub-postmasters – that the contractual terms set by Post Office regarding how they investigate and handle shortfalls are unfair. There would have been at least two more trials, had the parties not settled last week with Post Office Limited offering an apology and £58m in compensation. Of this, the vast majority will go towards legal costs and to the fund which bankrolled the litigation – leaving claimants lucky to get much more than £10k on average. Disappointing, sure, but better than nothing and that is what they could have got had the trials and inevitable appeals continued.

As would be expected for a trial depending on highly technical arguments, expert evidence featured heavily. The Post Office expert took a quantitative approach, presenting a statistical argument that claimant’s losses were implausibly high. This argument went by making a rough approximation as to the total losses of all sub-postmasters resulting from bugs in Horizon. Then, by assuming that these losses were spread over all sub-postmasters equally, losses by the 587 claimants would be no more than £25,000 – far less than the £18.7 million claimed. On this basis, the Post Office said that it is implausible for Horizon bugs to be the cause of the losses, and instead they are the fault of the affected sub-postmasters.

This argument is fundamentally flawed; I said so at the time, as did others. The claimant group was selected specifically as people who thought they were victims of Horizon bugs so it’s quite reasonable to think this group might indeed be disproportionally affected by Horizon bugs. The judge agreed, saying, “The group has a bias, in statistical terms. They plainly cannot be treated, in statistical terms, as though they are a random group of 587 [sub-postmasters]”. This error can be corrected, but the argument becomes circular and a statistical approach adds little new information. As the judgement concludes, “probability theory only takes one so far in this case, and that is not very far”.

So how can such cases be resolved? The expert witness for the claimants took a qualitative approach, investigating records of bugs but then making a subjective assessment as to how robust Horizon was, steadfastly refusing to be more specific under cross-examination. I can see why this is less satisfying, but it was the approach the judge preferred, and I think probably as good as one can do, given our limited understanding of how to statistically model the effect of software bugs. Had the trial not been settled, I’m not sure what would have happened next – sure, Horizon is not remotely robust, but this doesn’t tell us what actually happened in the 587 cases, each with their own unique characteristics.

This situation, however unsatisfactory, is still far better than the payment disputes I see in practice. In the Post Office trial, a large proportion of the eye-watering costs were spent on forcing Post Office Limited to disclose documentation on Horizon’s bugs, which technical experts could then summarise to the judge. Every payment system dispute I’ve seen has made do with bland assertions that the computer system is working properly, which is the presumption in English law. The Post Office trial demonstrates how wrong the presumption can be, that overturning it in litigation is so expensive that little compensation is left, and even then the resulting situation gives nobody much satisfaction.

I think there’s a strong case for changing the presumption that computers work correctly, putting the burden of proof on the system operator to show that it was. That won’t be enough though, because today’s computer systems generally cannot produce this sort of evidence. We can do better, and I’ve written before on how such evidence could be obtained but so far there’s insufficient incentive for companies to do so. The Post Office trial may give some encouragement, but I hope the likely follow-up investigations will call for action on the broader lessons the trial presents too.

Image by Sang Hyun Cho from Pixabay.