EFF Tells Court Defendants Must Be Allowed To Examine The DNA Software Used To Convict Them

Legal Issues

from the rolling-dice-with-more-sides-but-they're-still-just-dice dept

A proper adversarial system means the accused can confront the accuser. But that’s rarely the case when crime solving software is involved. The FBI doesn’t allow accused child porn downloaders to examine the malicious software it used to identify their computers. Multiple law enforcement agencies have dropped cases rather than discuss Stingray devices in open court.

All DNA analysis is handled by software. Most DNA analysis utilizes proprietary code created by private companies which license it to government agencies. The analysis may be performed by government agencies and employees, but when it comes to giving defense lawyers and their clients a chance to examine the software used to generate evidence, it suddenly becomes a very private matter.

Companies routinely intercede in criminal cases, telling judges that handing over source code or other information about their algorithms would somehow make it impossible for them to compete in the crime solving market. In most cases, judges are sympathetic to claims about trade secrets and proprietary code, allowing the accused to only confront their accuser by proxy, via a government expert or an employee of the software company.

In rare cases, the court actually finds in favor of the defendant. Earlier this year, a case involving third-party DNA software and the EFF’s intercession went the defendant’s way with a federal judge in Pennsylvania telling the government it couldn’t hide behind third-party trade secret assertions to keep this code out of the accused’s hands. As the court reasoned then, if DNA evidence is central to the case against the defendant, the defendant should have access to the evidence and the software that created it.

The EFF is hoping for a similar outcome in a case being handled in California. It deals with the possibly wrongful conviction of a 70-year-old man for rape. And it involves a DNA software company whose algorithm was the only one that tied the suspect to the crime.

An elderly woman was sexually assaulted and murdered in her home and two witnesses described seeing a black man in his 50s on the property on the day of the murder. Dozens of people had passed through the victim’s home in the few months leading up to the murder, including Mr. Davis and another individual. Mr. Davis is an African American man who was in his 70s at the time of the murder and suffers from Parkinson’s disease. Another individual who met the witnesses’ description had a history of sex crimes including sexual assault with a foreign object.

DNA samples were taken from dozens of locations and items at the crime scene. Mr. Davis’s DNA was not found on many of those, including a cane that was allegedly used to sexually assault the victim. Traditional DNA software was not able to match Mr. Davis to the DNA sample from a shoelace that was likely used to tie up the victim—but STRMix did, and the prosecution relied heavily on the latter before the jury.

As the EFF points out in its brief [PDF], DNA software is anything but infallible. STRMix was caught a half-decade ago when a bug in its code possibly led to dozens of false arrests and convictions. Presumably that bug has been patched, but if no one outside of STRMix is allowed to examine the code, it’s impossible to see if it might be leading prosecutors and government experts to overstate the certainty of DNA matches.

The necessity of independent source code review for probabilistic DNA programs was starkly demonstrated when FST (a counterpart to STRmix that was used in New York crime labs) was finally provided to a defense team for analysis. According to a defense expert, the undisclosed portion of the code could incorrectly tip the scales in favor of the prosecution’s hypothesis that a defendant’s DNA was present in a mixture. Reply Mem. of Law in Supp. as to Kevin Johnson at 19-21, United States v. Kevin Johnson, (S.D.N.Y. Feb. 27, 2017) (No. 15-CR-565 (VEC), D.I. 110). In fact, STRmix8 has suffered from programming errors that created false results in 60 cases in Queensland, Australia.

The problems caused by nondisclosure are especially acute in the context of the latest generation of probabilistic DNA analysis because there is no objective baseline truth against which the output from the program may be evaluated—and thus it is impossible to gauge the accuracy of these programs by examining their results.

If there’s no objective baseline, every DNA analysis program is allowed to grade on its own curve. DNA matches aren’t actually matches. They just reflect the likelihood of a match. With no baseline, the probability of it being an actual match is left to the discernment of prosecutors and their expert witnesses — all of whom come out looking better if they can secure a conviction.

Unlike breathalyzers, the latest generation of complex DNA analysis tools cannot be measured against an objective truth. Instead, these DNA programs are more akin to probabilistic election forecasting models, such as those designed by FiveThirtyEight and The Economist. The outputted results are based on the calculation of the probability of events—that the defendant, rather than a random person, contributed to the DNA mixture or that person X will win an election—a value that is not an objectively measurable fact. This is why different DNA programs, and even different laboratories using the same program, will generate substantially different results for the same sample.

This is why courts should allow defendants to examine the software that has, for the most part, accused them of committing crimes. If different algorithms produce different outcomes using the same inputs, none are to be trusted until they’re independently examined. And DNA software companies aren’t interested in that happening — not solely because of any trade secrets but because any defendant who successfully casts doubt on the accuracy of test results undermines their business model.

But protecting a business model isn’t the court’s business. The courts are there to serve justice, which means protecting the rights of the accused from accusers utilizing proprietary tech while waving around signed NDAs.

Filed Under: dna, due process, evidence, software